How to initialize each multithreading Pool worker with an individual value?

[Dan Stromberg]

On Tue, Nov 30, 2010 at 1:35 PM, Valery Khamenya <khamenya at gmail.com> wrote:
> Hi,
>
> multithreading.pool Pool has a promissing initializer argument in its
> constructor.
> However it doesn't look possible to use it to initialize each Pool's
> worker with some individual value (I'd wish to be wrong here)
>
> So, how to initialize each multithreading Pool worker with the
> individual values?
>
> The typical use case might be a connection pool, say, of 3 workers,
> where each of 3 workers has its own TCP/IP port.
>
> from multiprocessing.pool import Pool
>
> def port_initializer(_port):
>    global port
>    port = _port
>
> def use_connection(some_packet):
>    global _port
>    print "sending data over port # %s" % port
>
> if __name__ == "__main__":
>    ports=((4001,4002, 4003), )
>    p = Pool(3, port_initializer, ports) # oops... :-)
>    some_data_to_send = range(20)
>    p.map(use_connection, some_data_to_send)

Using an initializer with multiprocessing is something I've never tried.

I have used queues with multiprocessing though, and I believe you
could use them, at least as a fallback plan if you prefer to get the
initializer to work.

If you create in the parent a queue in shared memory (multiprocessing
facilitates this nicely), and fill that queue with the values in your
ports tuple, then you could have each child in the worker pool extract
a single value from this queue so each worker can have its own, unique
port value.

HTH