[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Antoine Pitrou solipsis at pitrou.net
Fri Oct 12 10:41:04 EDT 2018

On Fri, 12 Oct 2018 09:42:50 -0400
Sean Harrington <seanharr11 at gmail.com> wrote:
> I would contend that this is much more granular than Dask - this is just an
> optimization of Pool.map() to avoid redundantly passing the same `func`
> repeatedly, once per task, to each worker, with the primary goal of
> eliminating redundant serialization of large-memory-footprinted Callables.
> This is a different use case than Dask - I don't intend to approach the
> shared memory or distributed computing realms.
> And the second call to Pool.map would update the cached "self" as a part of
> its initialization workflow, s.t. "the latest version of self when map() is
> called is taken into account".

I still don't understand how that works.  If you "updated the cached
self", then surely you must transmit it to the child, right?



More information about the Python-Dev mailing list