[Numpy-discussion] Memmap with multiprocessing

Christoph Gohlke cgohlke at uci.edu
Wed Apr 27 18:07:43 EDT 2011


I don't think this was working correctly in numpy 1.4 either. The 
underlying problem seems to be that instance attributes of ndarray 
subtypes get lost during pickling:

import pickle
import numpy as np
class aarray(np.ndarray):
     def __new__(subtype):
         self = np.ndarray.__new__(subtype, (1,))
         self.attr = 'attr'
         return self
     def __array_finalize__(self, obj):
         self.attr = getattr(obj, 'attr', None)
a = aarray()
b = pickle.loads(a.dumps())
assert a.attr == b.attr, (a.attr, b.attr)

AssertionError: ('attr', None)

Christoph


On 4/26/2011 2:21 PM, Ralf Gommers wrote:
> On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes
> <totonixsame at gmail.com>  wrote:
>> Hi,
>>
>> Has anyone confirmed if this is a bug? Should I post this in the bug tracker?
>
> I see the same thing with recent master. Something very strange is
> going on in the memmap.__array_finalize__ method under Windows. Can
> you file a bug?
>
> Ralf
>
>
>>
>> Thanks!
>>
>> On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
>> <totonixsame at gmail.com>  wrote:
>>> Hi all,
>>>
>>> I'm having a error using memmap objects shared among processes created
>>> by the multprocessing module. This error only happen in Windows with
>>> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
>>> OS X it doesn't happen. This error is demonstrated by this little
>>> example script here https://gist.github.com/929168 , and the traceback
>>> is bellow (between<traceback>  tags):
>>>
>>> <traceback>
>>> Process Process-1:
>>> Traceback (most recent call last):
>>>   File "C:\Python26\Lib\multiprocessing\process.py", line 232, in _bootstrap
>>>     self.run()
>>>   File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
>>>     self._target(*self._args, **self._kwargs)
>>>   File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
>>> print_ma
>>> trix
>>>     print matrix
>>>   File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379, in
>>> arra
>>> y_str
>>>     return array2string(a, max_line_width, precision, suppress_small, ' ',
>>> "", s
>>> tr)
>>>   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 309, in
>>> ar
>>> ray2string
>>>     separator, prefix)
>>>   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 189, in
>>> _a
>>> rray2string
>>>     data = _leading_trailing(a)
>>>   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 162, in
>>> _l
>>> eading_trailing
>>>     min(len(a), _summaryEdgeItems))]
>>>   File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
>>> __arra
>>> y_finalize__
>>>     self.filename = obj.filename
>>> AttributeError: 'memmap' object has no attribute 'filename'
>>> Exception AttributeError: AttributeError("'NoneType' object has no attribute
>>> 'te
>>> ll'",) in<bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)>  ignored
>>> </traceback>
>>>
>>> I don't know if it's a bug, but I thought it's is import to report
>>> because the version 1.4.1 was working and 1.5.0 and above was not.
>>>
>>> Thanks!
>>>
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>



More information about the NumPy-Discussion mailing list