Multiprocessing, shared memory vs. pickled copies
robert.kern at gmail.com
Tue Apr 5 11:47:17 EDT 2011
On 4/4/11 7:05 PM, Robert Kern wrote:
> On 4/4/11 3:20 PM, John Ladasky wrote:
> However, at least in Python 2.7, multiprocessing seems to have a C extension
> module defining the Connection objects. Unfortunately, it looks like this C
> extension just imports the regular pickler that is not aware of these custom
> types. That's why you get this error. I believe this is a bug in Python.
> So what did you try, and what output did you get? What version of Python are you
>> I think that my understanding of multiprocessing
>> needs to evolve beyond the use of Pool, but I'm not sure yet. This
>> post suggests as much.
> Maybe. If the __reduce_ex__() method is implemented properly (and
> multiprocessing bugs aren't getting in the way), you ought to be able to pass
> them to a Pool just fine. You just need to make sure that the shared arrays are
> allocated before the Pool is started. And this only works on UNIX machines. The
> shared memory objects that shmarray uses can only be inherited. I believe that's
> what Sturla was getting at.
I'll take that back a little bit. Since the underlying shared memory types can
only be shared by inheritance, the ForkingPickler is only used for the arguments
passed to Process(), and not used for things sent through Queues (which Pool
uses). Since Pool cannot guarantee that the data exists before the Pool starts
its subprocesses, it must use the general mechanism.
So in short, if you pass the shmarrays as arguments to the target function in
Process(), it should work fine.
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
More information about the Python-list