Distributed Computing
Sage comes built in with a powerful distributed computing framework
called Distributed Sage (dsage).
Overview
Distributed Sage is a framework that allows one to do distributed
computing from within Sage. It includes a server, client and
workers as well as a set of classes that one can subclass from to
write distributed computation jobs. It is designed to be used
mainly for ‘coarsely’ distributed computations, i.e., computations
where jobs do not have to communicate much with each other. This is
also sometimes referred to as ‘grid’ computing.
There are 3 parts that make up Distributed Sage:
- The server is responsible for job distribution, submission
and collection. It also includes a web interface from which you can
monitor your jobs and do other administrative tasks.
- The client is responsible for submitting new jobs to the
server and collecting the results.
- The workers perform the actual computations.
Quick Start
Here are a few illustrations of how get up and running with
dsage.
Example 1
- Run dsage.setup(). This will setup the SQLite database and
generate a private and public key to be used for SSL communication.
It will also add a default user whose username defaults to your
current username.
- Run d = dsage.start_all(). This command will launch the
server, the web server,
workers and return an object
(d) which is a connection to the server. From here on, your
interaction with dsage will be mainly though the d object.
- Open up your browser and go to http://localhost:8082 to see the
web interface of dsage. From here you will be able to see the
status of your jobs, the workers connected and other important
information about your dsage server.
- Let’s begin with a simple example. Type job = d('2+2'). If you
look at the web interface, you should see that there is a new job
in the table. Now one of your workers will fetch that job, execute
it and present you the result. To get at the result, type
job.result. It might not be there yet because for this simple
computation, the network communication overhead dominates the
computational time. If you want to wait for your job to finish, you
can call job.wait() which will block until the job completes, at
which time you can inspect job.result for the result. You can do
any computation in this way by calling d.
Example 2
In this example, we will show you how to use the DistributedFactor
class that comes built-in with dsage. DistributedFactor attempts
to factor numbers by using a combination of the ECM and the QSieve
algorithm, as well as trial factorization for small factors.
- Run d = dsage.start_all() if you have not started your
dsage session yet, otherwise you can continue to use the previous
d instance.
- Start the distributed factoring job with
factor_job = DistributedFactor(d, number). You can pick fairly
large values, try for example
. You can see
whether or not the factoring job is done by looking at the
factor_job.done attribute. When it is done, you can look at the
prime factors it found by inspecting
factor_job.prime_factors.
Example 3
This example demonstrates a distributed version of the map
method. You can find the documentation for the regular map method
here: http://docs.python.org/lib/built-in-funcs.html. The syntax is
exactly the same.
First, run d = dsage.start_all() if you have not started your
dsage session yet; otherwise, you can continue to use the previous
d instance.
sage: def f(n): return n*n
sage: j = d.map(f, [25,12,25,32,12])
sage: jobs = d.block_on_jobs(j)
This will block until f has been evaluated for each of the inputs.
Example 4
This example demonstrates how to use the @parallel decorator with
a dsage instance.
First, run d = dsage.start_all() if you have not started your
dsage session yet; otherwise, you can continue to use the previous
d instance.
sage: P = parallel(p_iter = d.parallel_iter)
sage: @P
... def f(n,m): return n+m
sage: f([(1,2), (5, 10/3)])
Files
dsage stores a few files in $SAGE_ROOT/.sage/dsage:
- pubcert.pem and cacert.pem: the public and private keys the
server uses for SSL.
- dsage_key.pub and dsage_key: the keys used for
authenticating the user.
- The directory db/: this contains the dsage database.
- *.log files: logs generated by the server and workers.
- the directory tmp_worker_files/: workers store the jobs they
have processed here.