examples of realistic multiprocessing usage?

[TomF]

On 2011-01-16 20:57:41 -0800, Adam Skutt said:

> On Jan 16, 11:39 pm, TomF <tomf.sess... at gmail.com> wrote:
>> One difficulty is that there is a queue of work to be done and a queue
>> of results to be incorporated back into the parent; there is no
>> one-to-one correspondence between the two.  It's not obvious to me how
>> to coordinate the queues in a natural way to avoid deadlock or
>> starvation.
>> 
> 
> Depends on what you are doing.  If you can enqueue all the jobs before
> waiting for your results, then two queues are adequate.  The first
> queue is jobs to be accomplished, the second queue is the results.
> The items you put on the result queue have both the result and some
> sort of id so the results can be ordered after the fact.  Your parent
> thread of execution (thread hereafter) then:
> 
> 1. Adds jobs to the queue
> 2. Blocks until all the results are returned.  Given that you
> suggested that there isn't a 1:1 correspondence between jobs and
> results, have the queue support a message saying, 'Job X is done'.
> You're finished when all jobs send such a message.
> 3. Sorts the results into the desired ordered.
> 4. Acts on them.
> 
> If you cannot enqueue all the jobs before waiting for the results, I
> suggest turning the problem into a pipeline, such that the thread
> submitting the jobs and the thread acting on the results are
> different: submitter -> job processor -> results processor.
> Adam

Thanks for your reply.  I can enqueue all the jobs before waiting for 
the results, it's just that I want the parent to process the results as 
they come back.  I don't want the parent to block until all results are 
returned.  I was hoping the Pool module had a test for whether all 
processes were done, but I guess it isn't hard to keep track of that 
myself.

-Tom