We have a class which is a subclass of ndarray which defines __array_finalize__ to add an attribute. The class looks something like this:
class UnitArray(ndarray): # ... def __new__(cls, data, dtype=None, copy=True, units=None): # ... arr = array(data, dtype=dtype, copy=copy) res = ndarray.__new__(cls, arr.shape, arr.dtype,buffer=arr) res.units = units return res
def __array_finalize__(self, obj): if (isinstance(obj, UnitArray)): if hasattr(obj, 'units'): self.units = obj.units
This works for all the ufuncs I've tried, but the result of concatenate is missing the units attribute. Is this a bug with concatenate, or expected behavior?
The real class can be found here: https://svn.enthought.com/svn/enthought/trunk/src/lib/enthought/numerical_mo...
Bryce
On Tuesday 27 March 2007 20:08:04 Bryce Hendrix wrote:
We have a class which is a subclass of ndarray which defines __array_finalize__ to add an attribute. The class looks something like this:
Ahah, we ran into this problem a few months ago: You should not initialize your "units" attribute in the __new__ method, as it's not thread-safe first, but more important as you will lose it if you try to get a view of your new object (because in that case, __new__ is not called). Instead, try something like that:
def __array_finalize__(self, obj): self.units = getattr(obj, 'units', yourdefaultunit)
where your defaultunit is the value you would use in __new__. True, yourdefaultunit may change, and even be lost if there's a break between __new__ and __array_finalize__, but that should work. Lemme know.
Note that things are always tricky w/ concatenate: when you concatenate your UnitArray, you should have a UnitArray, OK. But if you mix UnitArray and ndarray ? What should take precedence ?
Thanks Pierre, works like a charm. One question though, how is defining a class attribute in __new__ any more thread-safe?
Bryce
Pierre GM wrote:
On Tuesday 27 March 2007 20:08:04 Bryce Hendrix wrote:
We have a class which is a subclass of ndarray which defines __array_finalize__ to add an attribute. The class looks something like this:
Ahah, we ran into this problem a few months ago: You should not initialize your "units" attribute in the __new__ method, as it's not thread-safe first, but more important as you will lose it if you try to get a view of your new object (because in that case, __new__ is not called). Instead, try something like that:
def __array_finalize__(self, obj): self.units = getattr(obj, 'units', yourdefaultunit)
where your defaultunit is the value you would use in __new__. True, yourdefaultunit may change, and even be lost if there's a break between __new__ and __array_finalize__, but that should work. Lemme know.
Note that things are always tricky w/ concatenate: when you concatenate your UnitArray, you should have a UnitArray, OK. But if you mix UnitArray and ndarray ? What should take precedence ? _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On Wednesday 28 March 2007 12:42:52 Bryce Hendrix wrote:
Thanks Pierre, works like a charm. One question though, how is defining a class attribute in __new__ any more thread-safe?
It's not, of course, and that's why it shouldn't be used. However, it's quite convenient and easier to use, and do you really need thread-safe objects ? In your example, if "yourdefaultunit" takes some simple value, you could use that value directly instead of the class attributes, which could be slightly messier to read but definitely thread-safe.
The second aspect about initialization is that when a ndarray is viewed as one of its subclasses, the actual memory space has already been allocated, so there's no call to __new__. Instead, you have to rely on __array_finalize__ to initialize the attributes specific to your subclass.
I spoke too soon, this code fails with the example you gave:
def test_concatenate(self): unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters)
# create another instance with different units. This instance # is never used, but calls __new__ with different units unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet)
new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) self.assertEqual(new_unit_ary.units, meters)
Any other ideas?
Bryce
Pierre GM wrote:
On Wednesday 28 March 2007 12:42:52 Bryce Hendrix wrote:
Thanks Pierre, works like a charm. One question though, how is defining a class attribute in __new__ any more thread-safe?
It's not, of course, and that's why it shouldn't be used. However, it's quite convenient and easier to use, and do you really need thread-safe objects ? In your example, if "yourdefaultunit" takes some simple value, you could use that value directly instead of the class attributes, which could be slightly messier to read but definitely thread-safe.
The second aspect about initialization is that when a ndarray is viewed as one of its subclasses, the actual memory space has already been allocated, so there's no call to __new__. Instead, you have to rely on __array_finalize__ to initialize the attributes specific to your subclass. _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On Thursday 29 March 2007 12:04:51 Bryce Hendrix wrote:
I spoke too soon, this code fails with the example you gave:
mmh, I tried to use the class you linked to last time: the only modifications I gave are listed below
class UnitArray(numpy.ndarray): __array_priority__ = 10.0 default_unit = "meters"
def __array_finalize__(self, obj): self.units = getattr(obj,'units',UnitArray.default_unit)
then I used that for testing: meters = "meters" feet = "feet" unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters)
unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet)
new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) print new_unit_ary.units
And it seems to work. Could you be a bit more specific ?
doh! I followed the example on the Wiki which does not define the class attribute in the class scope, but in __new__. Adding the declaration to the class scope seems to work.
Thanks, Bryce
Pierre GM wrote:
On Thursday 29 March 2007 12:04:51 Bryce Hendrix wrote:
I spoke too soon, this code fails with the example you gave:
mmh, I tried to use the class you linked to last time: the only modifications I gave are listed below
class UnitArray(numpy.ndarray): __array_priority__ = 10.0 default_unit = "meters"
def __array_finalize__(self, obj): self.units = getattr(obj,'units',UnitArray.default_unit)then I used that for testing: meters = "meters" feet = "feet" unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters)
unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet) new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) print new_unit_ary.unitsAnd it seems to work. Could you be a bit more specific ? _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On Thursday 29 March 2007 13:47:48 Bryce Hendrix wrote:
doh! I followed the example on the Wiki which does not define the class attribute in the class scope, but in __new__. Adding the declaration to the class scope seems to work.
Yeah, sorry about that, I really should update this wiki page. I gonna try later this afternoon/evening. Glad it works, though.
However, that only part of your problem: With concatenate, what unit should take precedence ? In your example, concatenating unit_ary_1 and unit_ary_2 should give a unit_array w/ meters as units. But what if you concatenate unit_1 and unit_3 ? Meters or feet ?
The idea would be to write a custom concatenate function, that would convert the units of each argument to the unit of the first one (for example), and call numpy.concatenate. Or something like that.
I really should widen my tests before proclaiming success... If you change the default units to "feet", the result of concatenating two UnitArrays instances with "meters" units is a UnitArray with "feet".
Bryce
Pierre GM wrote:
On Thursday 29 March 2007 12:04:51 Bryce Hendrix wrote:
I spoke too soon, this code fails with the example you gave:
mmh, I tried to use the class you linked to last time: the only modifications I gave are listed below
class UnitArray(numpy.ndarray): __array_priority__ = 10.0 default_unit = "meters"
def __array_finalize__(self, obj): self.units = getattr(obj,'units',UnitArray.default_unit)then I used that for testing: meters = "meters" feet = "feet" unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters)
unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet) new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) print new_unit_ary.unitsAnd it seems to work. Could you be a bit more specific ? _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Yeah, I think we may need to write some custom wrapper functions for the functions we use which don't call __new__. I don't really see another way around it since by the time __array_finalize__ gets called there are no units and the problems with mixed units.
bryce
Pierre GM wrote:
On Thursday 29 March 2007 13:47:48 Bryce Hendrix wrote:
doh! I followed the example on the Wiki which does not define the class attribute in the class scope, but in __new__. Adding the declaration to the class scope seems to work.
Yeah, sorry about that, I really should update this wiki page. I gonna try later this afternoon/evening. Glad it works, though.
However, that only part of your problem: With concatenate, what unit should take precedence ? In your example, concatenating unit_ary_1 and unit_ary_2 should give a unit_array w/ meters as units. But what if you concatenate unit_1 and unit_3 ? Meters or feet ?
The idea would be to write a custom concatenate function, that would convert the units of each argument to the unit of the first one (for example), and call numpy.concatenate. Or something like that. _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On Thursday 29 March 2007 14:09:38 Bryce Hendrix wrote:
I really should widen my tests before proclaiming success... If you change the default units to "feet", the result of concatenating two UnitArrays instances with "meters" units is a UnitArray with "feet".
Not a surprise at all, and I should have thought about it:
concatenate creates a plain ndarray first and ends up calling Unit_array.__array_finalize__. As the caller has no unit yet (it's a plain ndarray), the result gets the Unit_array.default_unit.
That's why you should really write a custom concatenate function that checks the units of its inputs, and sets the unit of the output accordingly.
Yup, we've decided to write custom concatenate & where methods. Thanks for all the help.
Bryce
Pierre GM wrote:
On Thursday 29 March 2007 14:09:38 Bryce Hendrix wrote:
I really should widen my tests before proclaiming success... If you change the default units to "feet", the result of concatenating two UnitArrays instances with "meters" units is a UnitArray with "feet".
Not a surprise at all, and I should have thought about it:
concatenate creates a plain ndarray first and ends up calling Unit_array.__array_finalize__. As the caller has no unit yet (it's a plain ndarray), the result gets the Unit_array.default_unit.
That's why you should really write a custom concatenate function that checks the units of its inputs, and sets the unit of the output accordingly. _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
-
Bryce Hendrix -
Pierre GM