One idea would be for the Pool method to generate a uuid and slap it on the
function as an attribute. If a function being passed in doesn't have one,
generate one. If it already has one, just pass that instead of pickling.
The child process will keep a cache mapping uuids to functions.
I'm still worried about unintended consequences.
On Thu, Oct 18, 2018 at 9:00 AM Michael Selik
On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington
wrote: The most common use case comes up when passing instance methods (of really big objects!) to Pool.map().
This reminds me of that old joke: "A patient says to the doctor, 'Doctor, it hurts when I ...!' The doctor replies, 'Well, don't do that.'"
Further, let me pivot on my idea of __qualname__...we can use the `id` of
`func` as the cache key to address your concern, and store this `id` on the `task` tuple (i.e. an integer in-lieu of the `func` previously stored there).
Possible. Does the Pool keep a reference to the passed function in the main process? If not, couldn't the garbage collector free that memory location and a new function could replace it? Then it could have the same qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to reproduce as it'd be dependent on the vagaries of memory allocation.