[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Antoine Pitrou]

Hi,

On Fri, 28 Sep 2018 17:07:33 -0400
Sean Harrington <seanharr11 at gmail.com> wrote:
> 
> In *short*, the implementation of the feature works as follows:
> 
>    1. Exposes a kwarg on Pool.__init__ called `expect_initret`, that
>    defaults to False. When set to True:
>       1. Capture the return value of the initializer kwarg of Pool
>       2. Pass this value to the function being applied, as a kwarg.
> 
> Again, in *short,* the motivation of the feature is to provide an explicit
> "flow of data" from parent process to worker process, and to avoid being
> *forced* to using the *global* keyword in initializer, or being *forced* to
> create global variables in the parent process.

Thanks for taking the time to explain your use case and write a
proposal.

My reactions to this are:

1. The proposed API is ugly.  This basically allows you to pass an
argument which changes with which arguments another function is later
called...
2. A global variable seems like the adequate way to represent a
process-global object (which is exactly your use case).
3. If you don't like globals, you could probably do something like
lazily-initialize the resource when a function needing it is executed;
this also avoids creating the resource if the child doesn't use it at
all.  Would that work for you?

As a more general remark, I understand the desire to make the Pool
object more flexible, but we can also not pile up features until it
satisfies all use cases.

As another general remark, concurrent.futures is IMHO the preferred API
for the future, and where feature work should probably concentrate.

Regards

Antoine.