[Tutor] When to use multiprocessing Managers?

[James Chapman]

Thanks for the explanation.

Is there any reason or case you can think of where on a single system
you would use the manager (proxied) queue over the multiprocessing
(piped) queue?

Actually I can probably answer this myself...

The manager could potentially be extended to do some kind of data
validation / error checking before submitting to the Queue. This could
be important if the data going into the Queue was for example, user
generated.

Hmm, yeah I'd say question answered. Thanks eryksun.


--
James


On 1 March 2014 16:48, eryksun <eryksun at gmail.com> wrote:
> On Fri, Feb 28, 2014 at 6:31 AM, James Chapman <james at uplinkzero.com> wrote:
>>
>> log_Q = multiprocessing.Queue()
>
> This is a Queue from multiprocessing.queues. It uses system resources
> (e.g. a semaphore for the queue capacity) that can be shared with
> processes on the same machine.
>
> A value `put` in a queue.Queue is available immediately:
>
>     >>> import queue
>     >>> q1 = queue.Queue()
>     >>> try: q1.put('value'); q1.get_nowait()
>     ... except queue.Empty: 'empty'
>     ...
>     'value'
>
> On the other hand, a Queue from multiprocessing.queues writes to a
> pipe using a background thread, so there can be a small delay:
>
>     >>> import multiprocessing as mp
>     >>> q2 = mp.Queue()
>     >>> try: q2.put('value'); q2.get_nowait()
>     ... except queue.Empty: 'empty'
>     ...
>     'empty'
>     >>> q2.get_nowait()
>     'value'
>
>> or whether I create it like this:
>>
>> multimanager = multiprocessing.Manager()
>> log_Q = multimanager.Queue()
>
> This is a queue.Queue wrapped by an AutoProxy. For example, its `get`
> method calls _callmethod('get', *args, **kwds), which connects to the
> manager, sends the request, and receives the result.
>
> The docs demonstrate using a manager with remote processes:
>
> http://docs.python.org/3/library/multiprocessing#using-a-remote-manager
>
> You can also proxy the Queue type from multiprocessing.queues. In that
> case, remote processes use a proxy, but local processes can use the
> queue directly.
>
>> Perhaps the manager would be important if I was writing to a Queue and
>> expecting all threads to see that message?
>
> Only 1 thread will `get` the message.