[Python-Dev] Forking and Multithreading - enemy brothers

[exarkun at twistedmatrix.com]

On 04:58 pm, jaedan31 at gmail.com wrote:
>Jesse Noller <jnoller <at> gmail.com> writes:
>>We already have an implementation that spawns a
>>subprocess and then pushes the required state to the child. The
>>fundamental need for things to be pickleable *all the time* kinda
>>makes it annoying to work with.
>
>This requirement puts a fairly large additional strain on working with
>unwieldy, wrapped C++ libraries in a multiprocessing environment.
>I'm not very knowledgeable on the internals of the system, but would
>it be possible to have some kind of fallback system whereby if an 
>object
>fails to pickle we instead send information about how to import it? 
>This
>has all kinds of limitations - it only works for importable things 
>(i.e. not
>instances), it can potentially lose information dynamically added to 
>the
>object, etc., but I thought I would throw the idea out there so someone
>knowledgeable can decide if it has any merit.

It's already possible to define pickling for arbitrary objects.  You 
should be able to do this for the kinds of importable objects you're 
talking about, and perhaps even for some of the actual instances (though 
that depends on how introspectable they are from Python, and whether the 
results of this introspection can be used to re-instantiate the object 
somewhere else).

Take a look at the copy_reg module.

Jean-Paul