<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#ffffff">
<br>
Hello Core Developers, <br>
<br>
My name is Dimitrios and I am newbie in python. I am working on a
Project (part of my PhD) that is called Synergeticprocessing module.
Initially is imitating the multiprocessing built in module but the
processes are distributed on a LAN and not Locally. The main issue I
have is with Pickle module. And I think I found some kind of BUG in
the built in multiprocessing module. <br>
<br>
(Synergeticprocessing module is located at GitHub: <a
class="moz-txt-link-freetext"
href="https://github.com/dpritsos/synergeticprocessing">https://github.com/dpritsos/synergeticprocessing</a>)<br>
<br>
Starting with the "BUG". In case someone uses the
multiprocessing.Pool of processes he/she has to face the problem of
types.MehtodType Impossible pickling. That is you cannot dispatch an
class instance method to the to the Process Pool. However, digging
in to the Source Code of the module there are few lines that resolve
this issue however are not activated or they are faultily activated
so they do not work. This is the 'BUG'<br>
<br>
<u>@ ....../multiprocessing/forking.py</u><br>
<br>
<br>
.<br>
.<br>
.<br>
<br>
#<br>
# Try making some callable types picklable<br>
#<br>
<br>
<font color="#ff0000">from pickle import Pickler<br>
class ForkingPickler(Pickler):<br>
dispatch = Pickler.dispatch.copy()<br>
<br>
@classmethod<br>
def register(cls, type, reduce):<br>
def dispatcher(self, obj):<br>
rv = reduce(obj)<br>
self.save_reduce(obj=obj, *rv)<br>
cls.dispatch[type] = dispatcher</font><br>
<br>
<font color="#33cc00">def _reduce_method(m):<br>
if m.im_self is None:<br>
return getattr, (m.im_class, m.im_func.func_name)<br>
else:<br>
return getattr, (m.im_self, m.im_func.func_name)<br>
ForkingPickler.register(type(ForkingPickler.save), _reduce_method)</font><br>
<br>
def _reduce_method_descriptor(m):<br>
return getattr, (m.__objclass__, m.__name__)<br>
<font color="#ff0000">ForkingPickler.register(type(list.append),
_reduce_method_descriptor)</font><br>
ForkingPickler.register(type(int.__add__),
_reduce_method_descriptor)<br>
<br>
#def _reduce_builtin_function_or_method(m):<br>
# return getattr, (m.__self__, m.__name__)<br>
#ForkingPickler.register(type(list().append),
_reduce_builtin_function_or_method)<br>
#ForkingPickler.register(type(int().__add__),
_reduce_builtin_function_or_method)<br>
.<br>
.<br>
.<br>
<br>
The <font color="#ff0000">RED</font> lines are not doing the job,
for some reason they are not managing to register the <font
color="#33cc00">GREEN</font> function as a global reduce/pickling
function even if you call the registration function into you
__main__ script. <br>
<br>
The solution I found is just to do this <br>
<b><br>
import copy_reg<br>
import types</b> <br>
<br>
<font color="#33cc00"><font color="#000000">def _reduce_method(m):<br>
if m.im_self is None:<br>
return getattr, (m.im_class, m.im_func.func_name)<br>
else:<br>
return getattr, (m.im_self, m.im_func.func_name)<br>
<br>
<b>copy_reg.pickle(types.MethodType, _reduce_method)</b><br>
.<br>
.<br>
.</font><br>
<br>
<font color="#000000">Doing that everything works FINE. But ONLY
for local methods i.e. the ones that their class is defined on
the __main__ script or other import-ed.<br>
<br>
In case you want to send something remotely (in an other
machine) or to an other __main__ script running separately then
you get a message like this:<br>
<br>
<font color="#ff0000">'module' object has no attribute
'<my_class>'</font><br>
<br>
The only way to resolve this is firstly to import a script that
has <my_class> defined there and everything works fine. <br>
<br>
SO the problems it seems to be that the <b>m.im_class</b>
(look code above) has some attribute __module__ defined as
__module__ = '__main__' or something like that. And this is the
reason why remote script cannot execute the function. I mean
that the _reduce_method() above DOES is pickling the whole CLASS
object so there is no reason not to be executed at the remote
script. Besides it does as mentioned above in you just import
this the user defined class form an other script. <br>
<br>
<br>
I have already spent about 12 weeks working on building my
synergeticPool and resolve the issue of Pickling and only 2 days
needed for the code of the Pool and the rest of the time was
spent for the Pickling issues, and study all the Class related
mechanics of python. That was the reason I ve started digging
the multipocessessing module and found this say 'BUG', and
finally sent this email. <br>
<br>
Best Regards, <br>
<br>
<br>
Dimitrios <br>
<br>
</font></font><br>
<br>
<br>
<br>
<br>
<br>
<br>
</body>
</html>