[Tutor] When to use multiprocessing Managers?

[eryksun]

On Fri, Feb 28, 2014 at 6:31 AM, James Chapman <james at uplinkzero.com> wrote:
>
> log_Q = multiprocessing.Queue()

This is a Queue from multiprocessing.queues. It uses system resources
(e.g. a semaphore for the queue capacity) that can be shared with
processes on the same machine.

A value `put` in a queue.Queue is available immediately:

    >>> import queue
    >>> q1 = queue.Queue()
    >>> try: q1.put('value'); q1.get_nowait()
    ... except queue.Empty: 'empty'
    ...
    'value'

On the other hand, a Queue from multiprocessing.queues writes to a
pipe using a background thread, so there can be a small delay:

    >>> import multiprocessing as mp
    >>> q2 = mp.Queue()
    >>> try: q2.put('value'); q2.get_nowait()
    ... except queue.Empty: 'empty'
    ...
    'empty'
    >>> q2.get_nowait()
    'value'

> or whether I create it like this:
>
> multimanager = multiprocessing.Manager()
> log_Q = multimanager.Queue()

This is a queue.Queue wrapped by an AutoProxy. For example, its `get`
method calls _callmethod('get', *args, **kwds), which connects to the
manager, sends the request, and receives the result.

The docs demonstrate using a manager with remote processes:

http://docs.python.org/3/library/multiprocessing#using-a-remote-manager

You can also proxy the Queue type from multiprocessing.queues. In that
case, remote processes use a proxy, but local processes can use the
queue directly.

> Perhaps the manager would be important if I was writing to a Queue and
> expecting all threads to see that message?

Only 1 thread will `get` the message.