Multiprocessing problem

Matt Chaput matt at whoosh.ca
Tue Mar 2 20:07:49 EST 2010


> If the main process doesn't get the results from the queue until the
> worker processes terminate, and the worker processes don't terminate
> until they've put their results in the queue, and the pipe consequently
> fills up, then deadlock can result.

The queue never fills up... on platforms with qsize() I can see this. I 
remove items from the results queue as I add to the job queue, and if I 
add timeouts everywhere the workers never raise Empty and the supervisor 
never raises Full. They just deadlock.

I've rewritten the code so the worker threads don't push information 
back while they run, they just write to a temporary file which the 
supervisor can read, which avoids the issue. But if anyone can tell me 
what I was doing wrong for future reference, I'd greatly appreciate it.

Thanks,

Matt



More information about the Python-list mailing list