subclassing from np.ndarray and np.rec.recarray
Hi -- We are subclassing from np.rec.recarray and are confused about how some methods of np.rec.recarray relate to (differ from) analogous methods of its parent, np.ndarray. Below are specific questions about the __eq__, __getitem__ and view methods, we'd appreciate answers to our specific questions and/or more general points that we may be not understanding about subclassing from np.ndarray (and np.rec.recarray). --- 1) Suppose I have a recarray object, x. How come np.ndarray.__getitem__(x, 'column_name') returns a recarray object rather than a ndarray? e.g., In [230]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [231]: np.ndarray.__getitem__(x, 'a') Out[231]: rec.array([1, 2]) In [232]: np.ndarray.__getitem__(x, 'a').dtype Out[232]: dtype('int32') The returned object is a recarray but it does not have a structured dtype. This generally seems to be the case when passing the instance of a subclass of np.ndarray (such as a np.rec.recarray object) to np.ndarray.__getitem__ --- 2)a) When I use the __getitem__ method of recarray to get an individual column, the returned object is an ndarray when the column is a numeric type but it is a recarray when the column is a string type. Why doesn't __getitem__ always return an ndarray for an individual column? e.g., In [175]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [176]: x['a'] Out[176]: array([1, 2]) In [177]: x['b'] Out[177]: rec.array(['dd', 'cc'], dtype='|S2') 2)b) Suppose I have a subclass of recarray, NewRecarray, that attaches some new attribute, e.g. 'info'. x = NewRecarray(data, names = ['a','b'], formats = '<i4, |S2') Now say I want to use recarray's __getitem__ method to get an individual column. Then x['a'] is an ndarray x['b'] is a NewRecarray and x['b'].info == x.info Is this the expected / proper behavior? Is there something wrong with the way I've subclassed recarray? --- 3)a) If I have two recarrays with the same len and column headers, the __eq__ method returns the rich comparison. Why is the result a recarray rather than an ndarray? In [162]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [163]: y = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [164]: x == y Out[164]: rec.array([ True, True], dtype=bool) 3)b) Suppose I have a subclass of recarray, NewRecarray, that attaches some new attribute, e.g. 'info'. x = NewRecarray(data) y = NewRecarray(data) z = x == y Then z is a NewRecarray object and z.info = x.info. Is this the expected / proper behavior? Is there something wrong with the way I've subclassed recarray? [Dan Yamins asked this a couple days ago] --- 4) Suppose I have a subclass of np.ndarray, NewArray, that attaches some new attribute, e.g. 'info'. When I view a NewArray object as a ndarray, the result has no 'info' attribute. Is the memory corresponding to the 'info' attribute garbage collected? What happens to it? x = NewArray(data) x.view(np.ndarray) has no 'info' attribute --- Thanks for any help! (And thanks for reading if you read any or all of this!) Elaine
On Jul 6, 2009, at 1:12 PM, Elaine Angelino wrote:
Hi -- We are subclassing from np.rec.recarray and are confused about how some methods of np.rec.recarray relate to (differ from) analogous methods of its parent, np.ndarray. Below are specific questions about the __eq__, __getitem__ and view methods, we'd appreciate answers to our specific questions and/or more general points that we may be not understanding about subclassing from np.ndarray (and np.rec.recarray).
For generic information about subclassing, please refer to: http://www.scipy.org/Subclasses http://docs.scipy.org/doc/numpy/user/basics.subclassing.html
1) Suppose I have a recarray object, x. How come np.ndarray.__getitem__(x, 'column_name') returns a recarray object rather than a ndarray? e.g.,
ndarray.__getitem__(x, item) calls x.__array_finalize__ if item is a basestring and not an integer. __array_finalize__ outputs an array of the same subtype as x (here, a recarray).
2)a) When I use the __getitem__ method of recarray to get an individual column, the returned object is an ndarray when the column is a numeric type but it is a recarray when the column is a string type. Why doesn't __getitem__ always return an ndarray for an individual column? e.g.,
In [175]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b'])
x.dtype
In your example. dtype([('a', '<i4'), ('b', '|S2')]) So, field 'a' has a dtype int, which is a built-in dtype, while field 'b' has a dtype '|S2', which is NOT a dtype. The code of recarray.__getitem__ shows you that in the first case, when the dtype of the output is a built-in, the output recarray (x['a']) is viewed as a standard ndarray. Not the case with x['b']. Why ? Ask Travis O.
2)b) Suppose I have a subclass of recarray, NewRecarray, that attaches some new attribute, e.g. 'info'.
x = NewRecarray(data, names = ['a','b'], formats = '<i4, |S2')
Now say I want to use recarray's __getitem__ method to get an individual column. Then
x['a'] is an ndarray x['b'] is a NewRecarray and x['b'].info == x.info
Is this the expected / proper behavior? Is there something wrong with the way I've subclassed recarray?
No, that's expected behavior. Once again, calling getitem with a field name as input calls __array_finalize__ internally. __array_finalize__ transforms the output in an array w/ the same subclass as your input: that's why x['b'] is a NewRecArray/ However, if the dtype of the output is builtin, it's transformed back to a standard ndarray: that's why x['a'] is a standard ndarray.
---
3)a) If I have two recarrays with the same len and column headers, the __eq__ method returns the rich comparison. Why is the result a recarray rather than an ndarray?
In [162]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [163]: y = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [164]: x == y Out[164]: rec.array([ True, True], dtype=bool)
OK, as far as I understand, here's what's going on: * First, we check whether the dtypes are compatible. * Then, each field of x is compared to the corresponding field of y, which calls a __array_finalize__ internally, and __array_wrap__ (because you call the 'equal' ufunc). * Then, a __array_finalize__ is called on the output, which transforms it back to a recarray.
3)b) Suppose I have a subclass of recarray, NewRecarray, that attaches some new attribute, e.g. 'info'.
x = NewRecarray(data) y = NewRecarray(data) z = x == y
Then z is a NewRecarray object and z.info = x.info.
Is this the expected / proper behavior? Is there something wrong with the way I've subclassed recarray? [Dan Yamins asked this a couple days ago]
To tell you whether there's something wrong, I'd need to see the code. I'm not especially surprised by this behavior...
---
4) Suppose I have a subclass of np.ndarray, NewArray, that attaches some new attribute, e.g. 'info'. When I view a NewArray object as a ndarray, the result has no 'info' attribute. Is the memory corresponding to the 'info' attribute garbage collected? What happens to it?
It's alive! No, seriously: when you take a view as a ndarray, you only access the portion of memory corresponding to the values of your ndarray and none of its extra info. Same thing as calling .__array__() on your object. So the information is still accessible, as long as the initial object exists (Correct me if I'm wrong on this one...)
participants (2)
-
Elaine Angelino
-
Pierre GM