New subject: Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues

16 Dec 2010

      Hello Core Developers,

My name is Dimitrios and I am newbie in python. I am working on a 
Project (part of my PhD) that is called Synergeticprocessing module. 
Initially is imitating the multiprocessing built in module but the 
processes are distributed on a LAN and not Locally. The main issue I 
have is with Pickle module. And I think I found some kind of BUG in the 
built in multiprocessing module.

(Synergeticprocessing module is located at GitHub: 
https://github.com/dpritsos/synergeticprocessing)

Starting with the "BUG". In case someone uses the multiprocessing.Pool 
of processes he/she has to face the problem of types.MehtodType 
Impossible pickling. That is you cannot dispatch an class instance 
method to the to the Process Pool. However, digging in to the Source 
Code of the module there are few lines that resolve this issue however 
are not activated or they are faultily activated so they do not work. 
This is the 'BUG'

_@ ....../multiprocessing/forking.py_

.
.
.

#
# Try making some callable types picklable
#

from pickle import Pickler
class ForkingPickler(Pickler):
     dispatch = Pickler.dispatch.copy()

     @classmethod
     def register(cls, type, reduce):
         def dispatcher(self, obj):
             rv = reduce(obj)
             self.save_reduce(obj=obj, *rv)
         cls.dispatch[type] = dispatcher

def _reduce_method(m):
     if m.im_self is None:
         return getattr, (m.im_class, m.im_func.func_name)
     else:
         return getattr, (m.im_self, m.im_func.func_name)
ForkingPickler.register(type(ForkingPickler.save), _reduce_method)

def _reduce_method_descriptor(m):
     return getattr, (m.__objclass__, m.__name__)
ForkingPickler.register(type(list.append), _reduce_method_descriptor)
ForkingPickler.register(type(int.__add__), _reduce_method_descriptor)

#def _reduce_builtin_function_or_method(m):
#    return getattr, (m.__self__, m.__name__)
#ForkingPickler.register(type(list().append), 
_reduce_builtin_function_or_method)
#ForkingPickler.register(type(int().__add__), 
_reduce_builtin_function_or_method)
.
.
.

The RED lines are not doing the job, for some reason they are not 
managing to register the GREEN function as a global reduce/pickling 
function even if you call the registration function into you __main__ 
script.

The solution I found is just to do this
*
import copy_reg
import types*

def _reduce_method(m):
     if m.im_self is None:
         return getattr, (m.im_class, m.im_func.func_name)
     else:
         return getattr, (m.im_self, m.im_func.func_name)

*copy_reg.pickle(types.MethodType, _reduce_method)*
.
.
.

Doing that everything works FINE. But ONLY for local methods i.e. the 
ones that their class is defined on the __main__ script or other import-ed.

In case you want to send something remotely (in an other machine) or to 
an other __main__ script running separately then you get a message like 
this:

'module' object has no attribute ''

The only way to resolve this is firstly to import a script that has 
 defined there and everything works fine.

SO the problems it seems to be that the *m.im_class*  (look code above) 
has some attribute __module__ defined as __module__ = '__main__' or 
something like that. And this is the reason why remote script cannot 
execute the function. I mean that the _reduce_method() above DOES is 
pickling the whole CLASS object so there is no reason not to be executed 
at the remote script. Besides it does as mentioned above in you just 
import this the user defined class form an other script.

I have already spent about 12 weeks working on building my 
synergeticPool and resolve the issue of Pickling and only 2 days needed 
for the code of the Pool and the rest of the time was spent for the 
Pickling issues, and study all the Class related mechanics of python. 
That was the reason I ve started digging the multipocessessing module 
and found this say 'BUG', and finally sent this email.

Best Regards,

Dimitrios

Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues

Dimitrios Pritsos

Michael Foord

Dimitrios Pritsos

Antoine Pitrou

exarkun＠twistedmatrix.com

tags

participants (4)