Multiprocessing, shared memory vs. pickled copies
robert.kern at gmail.com
Thu Apr 7 22:01:18 CEST 2011
On 4/7/11 1:39 PM, John Ladasky wrote:
> On Apr 7, 10:44 am, Robert Kern<robert.k... at gmail.com> wrote:
>> On 4/7/11 1:40 AM, John Ladasky wrote:
>>> On Apr 5, 10:43 am, Philip Semanchuk<phi... at semanchuk.com> wrote:
>>>> And as Robert Kern pointed out, numpy arrays are also pickle-able.
>>> OK, but SUBCLASSES of numpy.ndarray are not, in my hands, pickling as
>>> I would expect. I already have lots of code that is based on such
>>> subclasses, and they do everything I want EXCEPT pickle correctly. I
>>> may have to direct this line of questions to the numpy discussion
>>> group after all.
>> Define the __reduce_ex__() method, not __getstate__(), __setstate__().
>> ndarrays are extension types, so they use that mechanism.
> Thanks, Robert, as you can see, I got on that track shortly after I
> posted my code example. This is apparently NOT a numpy issue, it's an
> issue for pickling all C extension types.
Yes, but seriously, you should ask on the numpy mailing list. You will probably
run into more numpy-specific issues. At least, we'd have been able to tell you
things like "ndarray is an extension type, so look at that part of the
> Is there a way to examine a Python object, and determine whether it's
> a C extension type or not?
For sure? No, not really. Not at the Python level, at least. You may be able to
do something at the C level, I don't know.
> Or do you have to deduce that from the
> documentation and/or the source code?
> I started hunting through the numpy source code last night. It's
> complicated. I haven't found the ndarray object definition yet.
> Perhaps because I was looking through .py files, when I actually
> should have been looking through .c files?
Yes. The implementation for __reduce__ is in numpy/core/src/multiarray/methods.c
as array_reduce(). You may want to look in numpy/ma/core.py for the definition
of MaskedArray. It shows how you would define __reduce__, __getstate__ and
__setstate__ for a subclass of ndarray.
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
More information about the Python-list