70% [* SPAM *] Re: Re: multiprocessing.Queue blocks when sending large object

[DPalao]

El Lunes Diciembre 5 2011, boB at mail.python.org escribió:
> On Mon, 5 Dec 2011 09:02:08 +0100, DPalao <dpalao.python at gmail.com>
> 
> wrote:
> >El Martes Noviembre 29 2011, DPalao escribió:
> >> Hello,
> >> I'm trying to use multiprocessing to parallelize a code. There is a
> >> number of tasks (usually 12) that can be run independently. Each task
> >> produces a numpy array, and at the end, those arrays must be combined.
> >> I implemented this using Queues (multiprocessing.Queue): one for input
> >> and another for output.
> >> But the code blocks. And it must be related to the size of the item I
> >> put on the Queue: if I put a small array, the code works well; if the
> >> array is realistically large (in my case if can vary from 160kB to
> >> 1MB), the code blocks apparently forever.
> >> I have tried this:
> >> http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-
> >> lar ge- objects/
> >> but it didn't work (especifically I put a None sentinel at the end for
> >> each worker).
> >> 
> >> Before I change the implementation,
> >> is there a way to bypass this problem with  multiprocessing.Queue?
> >> Should I post the code (or a sketchy version of it)?
> >> 
> >> TIA,
> >> 
> >> David
> >
> >Just for reference. The other day I found the explanation by "ryles" on
> >his/her mail of 27th aug 2009, with title "Re: Q: multiprocessing.Queue
> >size limitations or bug...". It is very clarifying.
> >After having read that I arranged the program such that the main process
> >did not need to know when the others have finished, so I changed the
> >process join call with a queue get call, until a None (one per process)
> >is returned.
> >
> >Best,
> >
> >David
> 
> Why do people add character  like    [* SPAM *]  to their  subject
> lines ??   Is it supposed to do something  ??   I figured since
> programmers hang out here, maybe one of you know this.
> 
> Thanks,
> boB

Obviously it was not me who added the disgusting "70% [* SPAM *]" string to 
the subject. And I'd like to know the answer too.

David