Multiprocessing, shared memory vs. pickled copies

[Robert Kern]

On 4/7/11 1:39 PM, John Ladasky wrote:
> On Apr 7, 10:44 am, Robert Kern<robert.k... at gmail.com>  wrote:
>> On 4/7/11 1:40 AM, John Ladasky wrote:
>>
>>> On Apr 5, 10:43 am, Philip Semanchuk<phi... at semanchuk.com>    wrote:
>>>> And as Robert Kern pointed out, numpy arrays are also pickle-able.
>>
>>> OK, but SUBCLASSES of numpy.ndarray are not, in my hands, pickling as
>>> I would expect.  I already have lots of code that is based on such
>>> subclasses, and they do everything I want EXCEPT pickle correctly.  I
>>> may have to direct this line of questions to the numpy discussion
>>> group after all.
>>
>> Define the __reduce_ex__() method, not __getstate__(), __setstate__().
>>
>> http://docs.python.org/release/2.6.6/library/pickle.html#pickling-and...
>>
>> ndarrays are extension types, so they use that mechanism.
>
> Thanks, Robert, as you can see, I got on that track shortly after I
> posted my code example.  This is apparently NOT a numpy issue, it's an
> issue for pickling all C extension types.

Yes, but seriously, you should ask on the numpy mailing list. You will probably 
run into more numpy-specific issues. At least, we'd have been able to tell you 
things like "ndarray is an extension type, so look at that part of the 
documentation" quicker.

> Is there a way to examine a Python object, and determine whether it's
> a C extension type or not?

For sure? No, not really. Not at the Python level, at least. You may be able to 
do something at the C level, I don't know.

> Or do you have to deduce that from the
> documentation and/or the source code?
 >
> I started hunting through the numpy source code last night.  It's
> complicated.  I haven't found the ndarray object definition yet.
> Perhaps because I was looking through .py files, when I actually
> should have been looking through .c files?

Yes. The implementation for __reduce__ is in numpy/core/src/multiarray/methods.c 
as array_reduce(). You may want to look in numpy/ma/core.py for the definition 
of MaskedArray. It shows how you would define __reduce__, __getstate__ and 
__setstate__ for a subclass of ndarray.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco