Re: [Numpy-discussion] Attempting to sub-class NumArray
Todd Miller wrote:
I think the problem you are having is caused by the assumption that instance initialization is equivalent to an unordered assignment of all instance attributes. That's not true because the private attributes are interrelated; assigning "shape" affects "_shape" and "_strides", _itemsize must be known before _strides can be computed from shape, and so on.
It would be helpful if you could let us know what assignment ordering would be appropriate, or, if there were information about the interactions, perhaps we could work these things out for ourselves. As we've discussed before, there is little information available on private objects and their roles. Even to have the source code of some of the C code might be useful. I wasn't able to spot the relevant C code among the CVS directories. However, the assumption would appear to apply to the crude initialization of 'special'. The problem which is being reported seems to be connected with the function setshape in generic.py. This is far from the unordered assignment, although I recognize that ther might be some connection. Colin W.
On Fri, 2003-09-05 at 20:32, Colin J. Williams wrote:
Todd,
Thanks again for your prompt response. Todd Miller wrote:
Hi Colin,
I haven't looked at this in detail, but special.__init__() is making me queasy.
We don't call __init__ directly, but only when we construct the instance, a, of the class special.
Why doesn't it call NumArray.__init__()?
Because we have data to install in the newly created a. We go off to the factory function, numarray.array, to create the array, which we then specialize.
I don't see how NumArray.__init__ would help. This cumbersome procedure seems to work OK in most respects, but fails to display properly.
I don't follow the code sufficiently well to be sure, but my hunch is that the problem lies with setShape. It fails to update _strides.
Colin W.
Todd
On Fri, 2003-09-05 at 15:03, Colin J. Williams wrote:
When ravel() is used on an instance of NumArray, the _strides attribute is recalculated correctly. When ravel() is used on an instance of special, a sub-class of NumArray, the _strides attribute keeps its old value. This appears to cause the display problem which is reported.
The essentials of the code used are below.
To investigate this problem, two print statements were added to the method setshape in generic.py:
if newnelements == nelements: self._shape = tuple(shape) print 'In setShape self._shape:', self._shape self._strides = self._stridesFromShape() print 'In setShape self._strides:', self._strides else: raise ValueError("New shape is not consistent with the old shape")
It seems that only the first of these print statements is executed with special, but both are executed with NumArray. In the second case, the output is not in the expected sequence.
I would appreciate any suggestion as to a workaround.
Colin W,
Grief when attempting to sub-class NumArray
Code used:
# Try to sub-class class special(N.NumArray): def __init__(self, data= None, shape= None, eType= None): ## eType= _nt.Any or AnyType not acceptable %% arr= N.array(sequence= data, shape= shape, type= eType) for attr in dir(arr): ## This is a longwinded way of setting up special ## There must be a better way %% ## Perhaps we should use generic._view ## It's not documented TRY IT LATER ## There is no need to transfer methods! if attr[0] == '_' and attr[1] != '_': print attr exec 'self.' + attr + '= ' 'arr.' + attr exec 'print self.' + attr pass
a= special(data= [1, 2, 3, 4], shape= (2, 2)) a= N.array([1, 2, 3, 4], shape= (2, 2)) print 'In ts a._strides:', a._strides b= _gen.ravel(a) print 'In ts b._strides:', b._strides # <<< Unchanged with special, OK with NumArray print 'b:', b
C:\Progra~1\Python23\pythonw -u ts.py In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (8, 4) b: In setShape self._shape: (4,) Traceback (most recent call last): File "ts.py", line 37, in ?
USING THE SUBCLASS special print 'b:', b File "C:\PROGRA~1\Python23\lib\site-packages\numarray\numarraycore.py", line 618, in __str__ MAX_LINE_WIDTH, PRECISION, SUPPRESS_SMALL, ' ', "") File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 176, in array2string separator, prefix) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 138, in _array2string max_str_len = max(len(str(max_reduce(data))), libnumarray.error: maximum_Int32_reduce: access beyond buffer. offset=27 buffersize=16
Exit code: 1
# USING NumArray
In setShape self._shape: (2, 2) In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (4,) b: In setShape self._shape: (4,) [1 2 3 4]
Exit code: 0
------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
On Sat, 2003-09-06 at 11:25, Colin J. Williams wrote:
Todd Miller wrote:
I think the problem you are having is caused by the assumption that instance initialization is equivalent to an unordered assignment of all instance attributes. That's not true because the private attributes are interrelated; assigning "shape" affects "_shape" and "_strides", _itemsize must be known before _strides can be computed from shape, and so on.
It would be helpful if you could let us know what assignment ordering would be appropriate, or, if there were information about the interactions, perhaps we could work these things out for ourselves.
In truth, I don't think the order is the issue, so let's drop that. All I can say is that somehow special.__init__() is creating inconsistent _shape and _strides. I also noticed it creating and assigning bound methods, which you probably don't want to do.
As we've discussed before, there is little information available on private objects and their roles. Even to have the source code of some of the C code might be useful. I wasn't able to spot the relevant C code among the CVS directories.
The source code is all there. Look in Include/numarray/numarray.h, Src/_ndarraymodule.c, and Src/_numarraymodule.c, and maybe Src/libnumarraymodule.c.
However, the assumption would appear to apply to the crude initialization of 'special'. The problem which is being reported seems to be connected with the function setshape in generic.py. This is far from the unordered assignment, although I recognize that ther might be some connection.
I'm not sure where and how, but I think the problem is in your dir() loop. So, instead, try this: self.__setstate__(arr.__getstate__()) That ensures that you're only copying the data attributes and appears to work. Todd
Colin W.
On Fri, 2003-09-05 at 20:32, Colin J. Williams wrote:
Todd,
Thanks again for your prompt response. Todd Miller wrote:
Hi Colin,
I haven't looked at this in detail, but special.__init__() is making me queasy.
We don't call __init__ directly, but only when we construct the instance, a, of the class special.
Why doesn't it call NumArray.__init__()?
Because we have data to install in the newly created a. We go off to the factory function, numarray.array, to create the array, which we then specialize.
I don't see how NumArray.__init__ would help. This cumbersome procedure seems to work OK in most respects, but fails to display properly.
I don't follow the code sufficiently well to be sure, but my hunch is that the problem lies with setShape. It fails to update _strides.
Colin W.
Todd
On Fri, 2003-09-05 at 15:03, Colin J. Williams wrote:
When ravel() is used on an instance of NumArray, the _strides attribute is recalculated correctly. When ravel() is used on an instance of special, a sub-class of NumArray, the _strides attribute keeps its old value. This appears to cause the display problem which is reported.
The essentials of the code used are below.
To investigate this problem, two print statements were added to the method setshape in generic.py:
if newnelements == nelements: self._shape = tuple(shape) print 'In setShape self._shape:', self._shape self._strides = self._stridesFromShape() print 'In setShape self._strides:', self._strides else: raise ValueError("New shape is not consistent with the old shape")
It seems that only the first of these print statements is executed with special, but both are executed with NumArray. In the second case, the output is not in the expected sequence.
I would appreciate any suggestion as to a workaround.
Colin W,
Grief when attempting to sub-class NumArray
Code used:
# Try to sub-class class special(N.NumArray): def __init__(self, data= None, shape= None, eType= None): ## eType= _nt.Any or AnyType not acceptable %% arr= N.array(sequence= data, shape= shape, type= eType) for attr in dir(arr): ## This is a longwinded way of setting up special ## There must be a better way %% ## Perhaps we should use generic._view ## It's not documented TRY IT LATER ## There is no need to transfer methods! if attr[0] == '_' and attr[1] != '_': print attr exec 'self.' + attr + '= ' 'arr.' + attr exec 'print self.' + attr pass
a= special(data= [1, 2, 3, 4], shape= (2, 2)) a= N.array([1, 2, 3, 4], shape= (2, 2)) print 'In ts a._strides:', a._strides b= _gen.ravel(a) print 'In ts b._strides:', b._strides # <<< Unchanged with special, OK with NumArray print 'b:', b
C:\Progra~1\Python23\pythonw -u ts.py In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (8, 4) b: In setShape self._shape: (4,) Traceback (most recent call last): File "ts.py", line 37, in ?
USING THE SUBCLASS special print 'b:', b File "C:\PROGRA~1\Python23\lib\site-packages\numarray\numarraycore.py", line 618, in __str__ MAX_LINE_WIDTH, PRECISION, SUPPRESS_SMALL, ' ', "") File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 176, in array2string separator, prefix) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 138, in _array2string max_str_len = max(len(str(max_reduce(data))), libnumarray.error: maximum_Int32_reduce: access beyond buffer. offset=27 buffersize=16
Exit code: 1
# USING NumArray
In setShape self._shape: (2, 2) In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (4,) b: In setShape self._shape: (4,) [1 2 3 4]
Exit code: 0
------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- Todd Miller <jmiller@stsci.edu>
Todd Miller wrote:
On Sat, 2003-09-06 at 11:25, Colin J. Williams wrote:
Todd Miller wrote:
I think the problem you are having is caused by the assumption that instance initialization is equivalent to an unordered assignment of all instance attributes. That's not true because the private attributes are interrelated; assigning "shape" affects "_shape" and "_strides", _itemsize must be known before _strides can be computed from shape, and so on.
It would be helpful if you could let us know what assignment ordering would be appropriate, or, if there were information about the interactions, perhaps we could work these things out for ourselves.
In truth, I don't think the order is the issue, so let's drop that. All I can say is that somehow special.__init__() is creating inconsistent _shape and _strides. I also noticed it creating and assigning bound methods, which you probably don't want to do.
Yes, I had spotted that and limited the updating to the variables. This seemed to work, but your suggestion below is the neater way to go.
As we've discussed before, there is little information available on private objects and their roles. Even to have the source code of some of the C code might be useful. I wasn't able to spot the relevant C code among the CVS directories.
The source code is all there.
Look in Include/numarray/numarray.h, Src/_ndarraymodule.c, and Src/_numarraymodule.c, and maybe Src/libnumarraymodule.c.
Thanks, I'll poke around there.
However, the assumption would appear to apply to the crude initialization of 'special'. The problem which is being reported seems to be connected with the function setshape in generic.py. This is far from the unordered assignment, although I recognize that ther might be some connection.
I'm not sure where and how, but I think the problem is in your dir() loop. So, instead, try this:
self.__setstate__(arr.__getstate__())
This seems to be a better approach than mine.
That ensures that you're only copying the data attributes and appears to work.
Thanks, over that hurdle and on to the next Colin W.
Todd
Colin W.
On Fri, 2003-09-05 at 20:32, Colin J. Williams wrote:
Todd,
Thanks again for your prompt response. Todd Miller wrote:
Hi Colin,
I haven't looked at this in detail, but special.__init__() is making me queasy.
We don't call __init__ directly, but only when we construct the instance, a, of the class special.
Why doesn't it call NumArray.__init__()?
Because we have data to install in the newly created a. We go off to the factory function, numarray.array, to create the array, which we then specialize.
I don't see how NumArray.__init__ would help. This cumbersome procedure seems to work OK in most respects, but fails to display properly.
I don't follow the code sufficiently well to be sure, but my hunch is that the problem lies with setShape. It fails to update _strides.
Colin W.
Todd
On Fri, 2003-09-05 at 15:03, Colin J. Williams wrote:
When ravel() is used on an instance of NumArray, the _strides attribute is recalculated correctly. When ravel() is used on an instance of special, a sub-class of NumArray, the _strides attribute keeps its old value. This appears to cause the display problem which is reported.
The essentials of the code used are below.
To investigate this problem, two print statements were added to the method setshape in generic.py:
if newnelements == nelements: self._shape = tuple(shape) print 'In setShape self._shape:', self._shape self._strides = self._stridesFromShape() print 'In setShape self._strides:', self._strides else: raise ValueError("New shape is not consistent with the old shape")
It seems that only the first of these print statements is executed with special, but both are executed with NumArray. In the second case, the output is not in the expected sequence.
I would appreciate any suggestion as to a workaround.
Colin W,
Grief when attempting to sub-class NumArray
Code used:
# Try to sub-class class special(N.NumArray): def __init__(self, data= None, shape= None, eType= None): ## eType= _nt.Any or AnyType not acceptable %% arr= N.array(sequence= data, shape= shape, type= eType) for attr in dir(arr): ## This is a longwinded way of setting up special ## There must be a better way %% ## Perhaps we should use generic._view ## It's not documented TRY IT LATER ## There is no need to transfer methods! if attr[0] == '_' and attr[1] != '_': print attr exec 'self.' + attr + '= ' 'arr.' + attr exec 'print self.' + attr pass
a= special(data= [1, 2, 3, 4], shape= (2, 2)) a= N.array([1, 2, 3, 4], shape= (2, 2)) print 'In ts a._strides:', a._strides b= _gen.ravel(a) print 'In ts b._strides:', b._strides # <<< Unchanged with special, OK with NumArray print 'b:', b
USING THE SUBCLASS special
>C:\Progra~1\Python23\pythonw -u ts.py > > In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (8, 4) b: In setShape self._shape: (4,) Traceback (most recent call last): File "ts.py", line 37, in ? print 'b:', b File "C:\PROGRA~1\Python23\lib\site-packages\numarray\numarraycore.py", line 618, in __str__ MAX_LINE_WIDTH, PRECISION, SUPPRESS_SMALL, ' ', "") File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 176, in array2string separator, prefix) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 138, in _array2string max_str_len = max(len(str(max_reduce(data))), libnumarray.error: maximum_Int32_reduce: access beyond buffer. offset=27 buffersize=16
>Exit code: 1 > > __________________________________________________________
# USING NumArray
In setShape self._shape: (2, 2) In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (4,) b: In setShape self._shape: (4,) [1 2 3 4]
>Exit code: 0 > >
------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
participants (2)
-
Colin J. Williams -
Todd Miller