[Python-Dev] multiprocessing not compatible with functional.partial

Calvin Spealman ironfroggy at gmail.com
Thu Feb 12 15:22:41 CET 2009


On Thu, Feb 12, 2009 at 9:06 AM, Christian Heimes <lists at cheimes.de> wrote:
> Neal Becker schrieb:
>> If the argument to pool.map (f, args)
>> is
>> f = functional.partial (my_func, some_keyword_arg=whatever)
>>
>> I get:
>>
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>     self.run()
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>     self._target(*self._args, **self._kwargs)
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>     task = get()
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-
>>     return recv()
>> TypeError: type 'partial' takes at least one argument
>
> functool.partial instances are not picklable. You have to teach
> multiprocessing how to serialize a functool.partial instance.

I don't think it would be unreasonable to consider either 1) making
functools.partial picklable (I don't know how feasible this is) or 2)
having multiprocessing, specifically, handle more stdlib types that
are likely to be used here.

Of course, if we get into "this is an extended set of types safe for
multiprocessing", we are likely to see more problems between versions
as a more difficult backwards compat target. So, maybe both are more
harm than good?

-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy


More information about the Python-Dev mailing list