Matt Knox a écrit :

Jesse Noller <jnoller <at> gmail.com> writes:

...

We already have an implementation that spawns a subprocess and then pushes the required state to the child. The fundamental need for things to be pickleable *all the time* kinda makes it annoying to work with.

just a lurker here... but this topic hits home with me so thought I'd chime in. I'm a windows user and I would *love* to use multiprocessing a lot more because *in theory* it solves a lot of the problems I deal with very nicely (lot sof financial data number crunching). However, the pickling requirement makes it very very difficult to actually get any reasonably complex code to work properly with it.

A lot of the time the functions I want to call in the spawned processes are actually fairly self contained and don't need most of the environment of the parent process shoved into it, so it's annoying that it fails because some data I don't even need in the child process can't be pickled.

What about having an option to skip all the parent environment data pickling and require the user to manually invoke any imports that are needed in the target functions as the first step inside their target function?

for example...

def target_function(object_from_module_xyz): import xyz return object_from_module_xyz.do_something()

and if I forgot to import all the stuff necessary for the arguments being passed into my function to work, then it's my own problem.

Although maybe there is some obvious problem with this that I am not seeing.

Anyway, just food for thought.

- Matt

Hello I don't really get it there... it seems to me that multiprocessing only requires picklability for the objects it needs to transfer, i.e those given as arguments to the called function, and thsoe put into multiprocessing queues/pipes. Global program data needn't be picklable - on windows it gets wholly recreated by the child process, from python bytecode. So if you're having pickle errors, it must be because the "object_from_module_xyz" itself is *not* picklable, maybe because it contains references to unpicklable objets. In such case, properly implementing pickle magic methods inside the object should do it, shouldn't it ? Regards, Pascal