[Python-Dev] Forking and Multithreading - enemy brothers

[Pascal Chambon]

Matt Knox a écrit :
> Jesse Noller <jnoller <at> gmail.com> writes:
>
>   
>> We already have an implementation that spawns a
>> subprocess and then pushes the required state to the child. The
>> fundamental need for things to be pickleable *all the time* kinda
>> makes it annoying to work with.
>>
>>     
>
> just a lurker here... but this topic hits home with me so thought I'd chime
> in. I'm a windows user and I would *love* to use multiprocessing a lot more
> because *in theory* it solves a lot of the problems I deal with very nicely
> (lot sof financial data number crunching). However, the pickling requirement
> makes it very very difficult to actually get any reasonably complex code to
> work properly with it.
>
> A lot of the time the functions I want to call in the spawned processes are
> actually fairly self contained and don't need most of the environment of the
> parent process shoved into it, so it's annoying that it fails because some data
> I don't even need in the child process can't be pickled.
>
> What about having an option to skip all the parent environment data pickling
> and require the user to manually invoke any imports that are needed in the
> target functions as the first step inside their target function?
>
> for example...
>
> def target_function(object_from_module_xyz):
>     import xyz
>     return object_from_module_xyz.do_something()
>
> and if I forgot to import all the stuff necessary for the arguments being
> passed into my function to work, then it's my own problem.
>
> Although maybe there is some obvious problem with this that I am not seeing.
>
> Anyway, just food for thought.
>
> - Matt
>
>   

Hello

I don't really get it there... it seems to me that multiprocessing only 
requires picklability for the objects it needs to transfer, i.e those 
given as arguments to the called function, and thsoe put into 
multiprocessing queues/pipes. Global program data needn't be picklable - 
on windows it gets wholly recreated by the child process, from python 
bytecode.

So if you're having pickle errors, it must be because the 
"object_from_module_xyz" itself is *not* picklable, maybe because it 
contains references to unpicklable objets. In such case, properly 
implementing pickle magic methods inside the object should do it, 
shouldn't it ?

Regards,
Pascal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100204/6a965cfa/attachment.htm>