examples of realistic multiprocessing usage?

[Adam Skutt]

On Jan 16, 11:39 pm, TomF <tomf.sess... at gmail.com> wrote:
> One difficulty is that there is a queue of work to be done and a queue
> of results to be incorporated back into the parent; there is no
> one-to-one correspondence between the two.  It's not obvious to me how
> to coordinate the queues in a natural way to avoid deadlock or
> starvation.
>

Depends on what you are doing.  If you can enqueue all the jobs before
waiting for your results, then two queues are adequate.  The first
queue is jobs to be accomplished, the second queue is the results.
The items you put on the result queue have both the result and some
sort of id so the results can be ordered after the fact.  Your parent
thread of execution (thread hereafter) then:

1. Adds jobs to the queue
2. Blocks until all the results are returned.  Given that you
suggested that there isn't a 1:1 correspondence between jobs and
results, have the queue support a message saying, 'Job X is done'.
You're finished when all jobs send such a message.
3. Sorts the results into the desired ordered.
4. Acts on them.

If you cannot enqueue all the jobs before waiting for the results, I
suggest turning the problem into a pipeline, such that the thread
submitting the jobs and the thread acting on the results are
different: submitter -> job processor -> results processor.

Again though, the devil is in the details and without more details,
it's hard to suggest an explicit approach.  The simplest way to avoid
contention between two queues is to just remove it entirely (by
converting the processing to a single pipeline like I suggested).  If
that is not possible, then I suggest moving to pipes (or some other
form of I/O based IPC) and asynchronous I/O.  But I'd only do that if
I really couldn't write a pipeline.

Adam