multiprocessing question/error

Aaron Brady castironpi at gmail.com
Sat Jan 17 03:43:35 EST 2009


On Jan 16, 11:39 pm, Eduardo Lenz <l... at joinville.udesc.br> wrote:
> Hi,
>
> I was using the former processing package with python 2.5 with no problems.
> After switching to python 2.6.1 I am having some problems with the same code.
> The problem seems to be related to the fact that I am using Pool.map
> with a bounded method, since it is inside a class. To clarify a little bit,
> let me show some parts of the code ....

Hello,

> class Pygen3(self)....

I infer this was a typo.

>     ....
>     ....
>     ....
>     def calcula(self,indiv):
>         ....
>         ....
>         ....
>
>    def evaluate(self):
>      ....
>      ....
>     indiv = range(mult*self.popsize,(mult+1)*self.popsize)
>    pool = Pool(processes=nproc)
>    results = pool.map(self.calcula,indiv)

I infer the indentation drifted.

>    ...
>    ...
>
> the error is the following
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.6/threading.py", line 477, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> _handle_tasks
>     put(task)
> PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
> __builtin__.instancemethod failed
>
> Thanks for your help.

Checking on two typos above.  I have this working:

from multiprocessing import Process

class C(object):
    def m1( self ):
        p1 = Process(target=self.m2, args=('bob1',))
        p2 = Process(target=self.m2, args=('bob2',))
        p1.start()
        p2.start()
        p1.join()
        p2.join()
    def m2( self, name ):
        print 'hello', name

if __name__ == '__main__':
    c= C()
    c.m1()

But I get the same error as you with this:

from multiprocessing import Process, Pool

class C(object):
    def m1( self ):
        arg= range( 10 )
        pool= Pool( 5 )
        results = pool.map(self.m2,arg)
        print results
    def m2( self, name ):
        print 'hello', name

if __name__ == '__main__':
    c= C()
    c.m1()

I build a workaround with this:

from multiprocessing import Process, Pool

class C(object):
    def m1( self ):
        arg= zip( [ self ]* 10, range( 10 ) )
        pool= Pool( 5 )
        results = pool.map(C_m2,arg)
        print results
    def m2( self, arg ):
        print 'hello', arg
        return arg* 2
def C_m2( ar, **kwar ):
    return C.m2( *ar, **kwar )

if __name__ == '__main__':
    c= C()
    c.m1()

Note that you have to explicitly include 'self' as a first parameter
when packing the arguments, and then explicitly unpack it in the
helper function.  Also, wrapping 'm2' with '@staticmethod' also fails
with this error:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\programs\python26\lib\threading.py", line 522, in
__bootstrap_inner
    self.run()
  File "c:\programs\python26\lib\threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "c:\programs\python26\lib\multiprocessing\pool.py", line 225,
in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

If this works for you, it's possible to make the helper function more
dynamic, by accepting a class name and an attribute name as string
arguments, then retrieving them in the subprocesses.



More information about the Python-list mailing list