[Python-Dev] multiprocessing not compatible with functional.partial

[Jesse Noller]

On Thu, Feb 12, 2009 at 9:22 AM, Calvin Spealman <ironfroggy at gmail.com> wrote:
> On Thu, Feb 12, 2009 at 9:06 AM, Christian Heimes <lists at cheimes.de> wrote:
>> Neal Becker schrieb:
>>> If the argument to pool.map (f, args)
>>> is
>>> f = functional.partial (my_func, some_keyword_arg=whatever)
>>>
>>> I get:
>>>
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>>     self.run()
>>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>>     self._target(*self._args, **self._kwargs)
>>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>>     task = get()
>>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>>     return recv()
>>> TypeError: type 'partial' takes at least one argument
>>
>> functool.partial instances are not picklable. You have to teach
>> multiprocessing how to serialize a functool.partial instance.
>
> I don't think it would be unreasonable to consider either 1) making
> functools.partial picklable (I don't know how feasible this is) or 2)
> having multiprocessing, specifically, handle more stdlib types that
> are likely to be used here.
>
> Of course, if we get into "this is an extended set of types safe for
> multiprocessing", we are likely to see more problems between versions
> as a more difficult backwards compat target. So, maybe both are more
> harm than good?

While maintaining a back port which contains all of the current and
future functionality is an admirable goal, the fact is is that over
time, there will simply be features added which will only work on
current+future, and not be able to be back ported.

Case in point, I want to look into adding a lot more contextmanager
support into the module - this can work back to 2.5, but not further
than that.

.02 cents.

-jesse