Multiprocessing.Queue - I want to end.

[Hendrik van Rooyen]

 "Luis Zarrabeitia" <akakyrie at uh.cu> wrote:

8< -------explanation and example of one producer, --------
8< -------more consumers and one queue --------------------

>As you can see, I'm sending one 'None' per consumer, and hoping that no 
>consumer will read more than one None. While this particular implementation 

You don't have to hope. You can write the consumers that way to guarantee it.

>ensures that, it is very fragile. Is there any way to signal the consumers? 

Signalling is not easy - you can signal a process, but I doubt if it is 
possible to signal a thread in a process.

>(or better yet, the queue itself, as it is shared by all consumers?) 
>Should "close" work for this? (raise the exception when the queue is 
>exhausted, not when it is closed by the producer).

I haven't the foggiest if this will work, and it seems to me to be kind
of involved compared to passing a sentinel or sentinels.

And while we are on the subject - Passing None as a sentinel is IMO as
good as or better than passing "XXZulu This is the End uluZXX",
or any other imaginative string that is not likely to occur naturally
in the input.

I have always wondered why people do the one queue many getters thing.

Given that the stuff you pass is homogenous in that it will require a
similar amount of effort to process, is there not a case to be made
to have as many queues as consumers, and to round robin the work?

And if the stuff you pass around needs disparate effort to consume,
it seems to me that you can more easily balance the load by having
specialised consumers, instead of instances of one humungous 
"I can eat anything" consumer.

I also think that having a queue per consumer thread makes it easier
to replace the threads with processes and the queues with pipes or
sockets if you need to do serious scaling later.

In fact I happen to believe that anything that does any work needs 
one and only one input queue and nothing else, but I am peculiar
that way.

- Hendrik