Ok. The thing I am trying to design/code is a job distribution system, for a distributed computation model. There is a server which interacts with the job database (which is just a bsddb). The server accepts connections from clients and distributes jobs to them. The server also accepts connections from a 2nd type of client, which submits jobs to the server.

Sounds like a distributed task queue.

Users/user processes submit tasks to the master queue manager (the "server") via a defined interface (HTTP post, SOAP, XML-RPC, PB, or ???). The master queue manager adds the requested tasks to the master job queue.

A secondary queue manager on each "client" machine (the "monitor") requests tasks from the master queue when its local queue drops below a defined threshold number of jobs (or drops to zero). When the local queue is empty, the secondary queue manager periodically requests tasks from the master. It adds any tasks received to its local queue.

For each received task, the secondary queue manager starts a worker task, immediately receiving a deferred. It registers success and error callbacks on the deferred.

When it gets the success or error callback from a worker, the secondary queue manager records the result and reports it to the master queue controller (the "server"). When it has successfully reported to the master, it deletes the task from the local queue.

Additional wrinkles: (1) Intertask dependencies may require that some tasks be performed sequentially, on the same slave machine, or both. (2) Some tasks may be more important than others, and so must be processed (or at least started) first. (3) Task cancellation at the master may require that the master be able to send asynchronous task-cancel instructions to the slaves.

The workers of course have to submit their results back to the server, but in the system I envisioned the monitor takes care of that by continuously polling the status of the workers.

If the worker returns a deferred, the monitor doesn't need to poll; it'll get a callback when the worker completes (or fails).

HTH.

- Sam