Multiprocessing, shared memory vs. pickled copies
ladasky at my-deja.com
Mon Apr 4 22:20:16 CEST 2011
I'm developing some custom neural network code. I'm using Python 2.6,
Numpy 1.5, and Ubuntu Linux 10.10. I have an AMD 1090T six-core CPU,
and I want to take full advantage of it. I love to hear my CPU fan
running, and watch my results come back faster.
When I'm training a neural network, I pass two numpy.ndarray objects
to a function called evaluate. One array contains the weights for the
neural network, and the other array contains the input data. The
evaluate function returns an array of output data.
I have been playing with multiprocessing for a while now, and I have
some familiarity with Pool. Apparently, arguments passed to a Pool
subprocess must be able to be pickled. Pickling is still a pretty
vague progress to me, but I can see that you have to write custom
__reduce__ and __setstate__ methods for your objects. An example of
code which creates a pickle-friendly ndarray subclass is here:
Now, I don't know that I actually HAVE to pass my neural network and
input data as copies -- they're both READ-ONLY objects for the
duration of an evaluate function (which can go on for quite a while).
So, I have also started to investigate shared-memory approaches. I
don't know how a shared-memory object is referenced by a subprocess
yet, but presumably you pass a reference to the object, rather than
the whole object. Also, it appears that subprocesses also acquire a
temporary lock over a shared memory object, and thus one process may
well spend time waiting for another (individual CPU caches may
sidestep this problem?) Anyway, an implementation of a shared-memory
ndarray is here:
I've added a few lines to this code which allows subclassing the
shared memory array, which I need (because my neural net objects are
more than just the array, they also contain meta-data). But I've run
into some trouble doing the actual sharing part. The shmarray class
CANNOT be pickled. I think that my understanding of multiprocessing
needs to evolve beyond the use of Pool, but I'm not sure yet. This
post suggests as much.
I don't believe that my questions are specific to numpy, which is why
I'm posting here, in a more general Python forum.
When should one pickle and copy? When to implement an object in
shared memory? Why is pickling apparently such a non-trivial process
anyway? And, given that multi-core CPU's are apparently here to stay,
should it be so difficult to make use of them?
More information about the Python-list