[execnet-dev] pickling issue using multiprocessing "Can't pickle ... it's not found as __channelexec__.f"

[Ronny Pfannschmidt]

Hi Ward,

when you remote_exec a module, it gets __name__ == "__channelexec__" and
is transient

therefore you cant import from it
which unfortunately breaks pickle loading of functions (which relies on
importing)

Is it necessary to mix execnet and multiprocessing?
it seems weird to use both to me. 

-- Ronny

On Thu, 2011-05-19 at 12:23 +0200, Ward vanaerschot wrote:
> Hi all,
> 
> 
> my original problem was this: run python code "Y" remotely which
> imports a module "X". On every channel.remote_exec("Y") the path to
> import module "X" on the remote side changes and contains a 'slightly'
> different version of X.py . However, since the server spawns several
> threads the consequence is that once a module "X" is loaded it will
> not be loaded again, which is not what I want.
> My original solution is to startup a "process" in "Y" which imports
> "X" and solves .
> The multiprocessing module seems convenient since I can pass Queue
> objects to it to facilitate the communication:
> 
> 
> in "Y":
> import sys, os
> from functools import partial
> import multiprocessing
> 
> 
> def f(receiveQ):
>    sys.path.append(os.path.join(pathToX)) # pathToX is assigned before
> this code fragment
>    import X
> 
> 
> def go()
>    multiprocessing.Process(target=partial(f, receiveQ))
> 
> 
> if __name__ == '__channelexec__':
>    go()
> 
> 
> I get the following trace:
> 
> 
> Traceback (most recent call last):
>   File "c:\python27\lib\threading.py", line 530, in __bootstrap_inner
>     self.run()
>   File "", line 226, in run
>   File "c:\python27\lib\multiprocessing\process.py", line 104, in
> start
>     self._popen = Popen(self)
>   File "c:\python27\lib\multiprocessing\forking.py", line 244, in
> __init__
>     dump(process_obj, to_child, HIGHEST_PROTOCOL)
>   File "c:\python27\lib\multiprocessing\forking.py", line 167, in dump
>     ForkingPickler(file, protocol).dump(obj)
>   File "c:\python27\lib\pickle.py", line 224, in dump
>     self.save(obj)
>   File "c:\python27\lib\pickle.py", line 331, in save
>     self.save_reduce(obj=obj, *rv)
>   File "c:\python27\lib\pickle.py", line 419, in save_reduce
>     save(state)
>   File "c:\python27\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "c:\python27\lib\pickle.py", line 649, in save_dict
>     self._batch_setitems(obj.iteritems())
>   File "c:\python27\lib\pickle.py", line 681, in _batch_setitems
>     save(v)
>   File "c:\python27\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "c:\python27\lib\pickle.py", line 748, in save_global
>     (obj, module, name))
> PicklingError: Can't pickle <function f at 0x0537B1F0>: it's not foun
> d as __channelexec__.f
> 
> 
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "c:\python27\lib\multiprocessing\forking.py", line 347, in main
>     self = load(from_parent)
>   File "c:\python27\lib\pickle.py", line 1378, in load
>     return Unpickler(file).load()
>   File "c:\python27\lib\pickle.py", line 858, in load
>     dispatch[key](self)
>   File "c:\python27\lib\pickle.py", line 880, in load_eof
>     raise EOFError
> EOFError
> 
> 
> 
> 
> Somehow pickle expects my function 'f' to 'live' in __channelexec__
> but it does not find it.
> I suppose this is because __channelexec__ is defined when executing
> the code but the code is compiled without this knowledge. 
> Is there a bug in gateway_base.SlaveGateway.executetask() which
> compiles and executes the source such that pickling is not possible?
> 
> 
> thanks!
> 
> 
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110519/d3cc9527/attachment.pgp>