[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

One idea would be for the Pool method to generate a uuid and slap it on the
function as an attribute. If a function being passed in doesn't have one,
generate one. If it already has one, just pass that instead of pickling.
The child process will keep a cache mapping uuids to functions.

I'm still worried about unintended consequences.


On Thu, Oct 18, 2018 at 9:00 AM Michael Selik <michael.selik at gmail.com>
wrote:

> On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington <seanharr11 at gmail.com>
> wrote:
>
>> The most common use case comes up when passing instance methods (of
>> really big objects!) to Pool.map().
>>
>
> This reminds me of that old joke: "A patient says to the doctor, 'Doctor,
> it hurts when I ...!' The doctor replies, 'Well, don't do that.'"
>
> Further, let me pivot on my idea of __qualname__...we can use the `id` of
>> `func` as the cache key to address your concern, and store this `id` on the
>> `task` tuple (i.e. an integer in-lieu of the `func` previously stored
>> there).
>>
>
> Possible. Does the Pool keep a reference to the passed function in the
> main process? If not, couldn't the garbage collector free that memory
> location and a new function could replace it? Then it could have the same
> qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to
> reproduce as it'd be dependent on the vagaries of memory allocation.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/4d2b7599/attachment.html>