On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe wrote:

NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.

The current approach is at a dead end, so something better needs to be done.

Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:

return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)

To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like.

The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default.

The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.

How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes that don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.

Thanks, Mark

A simple class which overrides the ufuncs might look as follows:

def sin(ufunc, *args, **kwargs): # Convert degrees to radians args[0] = np.deg2rad(args[0]) # Return a regular array, since the result is not in degrees return ufunc(*args, **kwargs)

class MyDegreesClass: """Array-like object with a degrees unit"""

def __init__(arr): self.arr = arr

def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: raise TypeError, 'ufunc %s incompatible with MyDegreesClass' % ufunc.name

A more complex example will be something like this:

def my_general_ufunc(ufunc, *args, **kwargs): # Extract the 'out' argument. This only supports ufuncs with # one output, currently. out = kwargs.get('out') if len(args) > ufunc.nin: if out is None: out = args[ufunc.nin] else: raise ValueError, "'out' given as both a position and keyword argument"

# Just want the inputs from here on args = args[:ufunc.nin]

# Strip out MyArrayClass, but allow operations with regular ndarrays raw_in = [] for a in args: if isinstance(a, MyArrayClass): raw_in.append(a.arr) else: raw_in.append(a)

# Allocate the output array if not out is None: if isinstance(out, MyArrayClass): raise TypeError, "'out' must have type MyArrayClass" else: # Create the output array, obeying the 'order' parameter, # but disallowing subclasses out = np.broadcast_empty_like([args, order=kwargs.get('order'), dtype=ufunc.result_type(args), subok=False)

# Override the output argument kwargs['out'] = out.arr

# Call the ufunc ufunc(*args, **kwargs)

# Return the output return out

class MyArrayClass: def __init__(arr): self.arr = arr

def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: return my_general_ufunc(ufunc, *args, **kwargs)

Chuck

On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe wrote:

On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:

...

On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe wrote:

...

NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.

The current approach is at a dead end, so something better needs to be done.

...

Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:

return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)

To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like.

The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default.

The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.

How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes that don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.

The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs. Supporting objects that aren't ndarray subclasses is one of the purposes for this approach, and neither of my two example cases subclassed ndarray.

Sounds good. Many of the current uses of __array_wrap__ that I am aware of are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled? <snip> Chuck

On Tue, Jun 21, 2011 at 12:46 PM, Darren Dale wrote:

On Tue, Jun 21, 2011 at 2:28 PM, Charles R Harris wrote:

...

On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe wrote:

...

On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris wrote:

...

On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe

...

...

...

...

NumPy has a mechanism built in to allow subclasses to adjust or

override

...

aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review

...

...

...

...

design to see if it meets your needs.

The current approach is at a dead end, so something better needs to be done.

...

Any class type which would like to override its behavior in ufuncs

would

...

define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the

...

...

...

...

battle gets its _numpy_ufunc_ function called as follows:

return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)

To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like. The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to

...

...

...

...

ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default. The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for

...

...

...

...

output, instead of just one.

How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes

...

...

...

don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.

The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False'

wrote: this priority the producing the that parameter

...

...

to the kwargs. Supporting objects that aren't ndarray subclasses is one of the purposes for this approach, and neither of my two example cases subclassed ndarray.

Sounds good. Many of the current uses of __array_wrap__ that I am aware of are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled?

I contributed the __array_prepare__ method a while back so classes could raise errors before the array data is modified in place. Specifically, I was concerned about units support in my quantities package (http://pypi.python.org/pypi/quantities). But I agree that this approach is needs to be reconsidered. It would be nice for subclasses to have an opportunity to intercept and process the values passed to a ufunc on their way in. For example, it would be nice if when I did np.cos(1.5 degrees), my subclass could intercept the value and pass a new one on to the ufunc machinery that is expressed in radians. I thought PJ Eby's generic functions PEP would be a really

Link to PEP-3124 http://tinyurl.com/3brnk6. Chuck

On Tue, Jun 21, 2011 at 1:46 PM, Darren Dale wrote:

On Tue, Jun 21, 2011 at 2:28 PM, Charles R Harris wrote:

...

On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe wrote:

...

On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris wrote:

...

On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe

...

...

...

...

NumPy has a mechanism built in to allow subclasses to adjust or

override

...

aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review

...

...

...

...

design to see if it meets your needs.

The current approach is at a dead end, so something better needs to be done.

...

Any class type which would like to override its behavior in ufuncs

would

...

define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the

...

...

...

...

battle gets its _numpy_ufunc_ function called as follows:

return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)

To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like. The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to

...

...

...

...

ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default. The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for

...

...

...

...

output, instead of just one.

How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes

...

...

...

don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.

The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False'

wrote: this priority the producing the that parameter

...

...

to the kwargs. Supporting objects that aren't ndarray subclasses is one of the purposes for this approach, and neither of my two example cases subclassed ndarray.

Sounds good. Many of the current uses of __array_wrap__ that I am aware of are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled?

I contributed the __array_prepare__ method a while back so classes could raise errors before the array data is modified in place. Specifically, I was concerned about units support in my quantities package (http://pypi.python.org/pypi/quantities). But I agree that this approach is needs to be reconsidered. It would be nice for subclasses to have an opportunity to intercept and process the values passed to a ufunc on their way in. For example, it would be nice if when I did np.cos(1.5 degrees), my subclass could intercept the value and pass a new one on to the ufunc machinery that is expressed in radians. I thought PJ Eby's generic functions PEP would be a really good way to handle ufuncs, but the PEP has stagnated.

I made one of my examples overriding sin with degrees, because I think this overloading method can work well for a physical quantities library. -Mark

Darren _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe wrote:

On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris wrote:

...

How does the ufunc get called so it doesn't get caught in an endless loop?

[...]

The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.

I didn't understand, could you please expand on that or show an example? Darren

On Wed, Jun 22, 2011 at 5:08 AM, Pauli Virtanen wrote:

Tue, 21 Jun 2011 16:43:13 -0500, Mark Wiebe wrote: [clip: __array_wrap__]

...

Those could stay as they are, and just the ufunc usage of __array_wrap__ can be deprecated. For classes which currently use __array_wrap__, they would just need to also implement _numpy_ufunc_ to eliminate any deprecation messages.

Do you mean that the new mechanism would not be able to do the same thing here?

It would have to be generalized a bit more to support these usages, because some functions produce outputs with different shapes, and the inputs may not be broadcast together in the same manner as in the element-wise ufuncs.

Preservation of array subclasses in linalg functions is not very uniform, and will likely need fixes in several of the functions. Since new code in any case would need to be written, I'd prefer using the "new" approach and so leaving us the option of marking the "old" approach deprecated.

I think creating a @ufunc_overload(...) decorator for specifying the ufunc properties like nin and nout might be a nice way to generalize it. -Mark

Pauli

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Jun 22, 2011 at 1:31 PM, Mark Wiebe wrote:

On Wed, Jun 22, 2011 at 7:34 AM, Lluís wrote:

...

Darren Dale writes:

...

On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe wrote:

...

On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris wrote:

...

How does the ufunc get called so it doesn't get caught in an endless loop?

...

[...]

...

...

The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.

...

I didn't understand, could you please expand on that or show an example?

As I understood the initial description and examples, the ufunc overload will keep being used as long as its arguments are of classes that declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" attribute).

Thus Mark's comment saying that you have to either transform the arguments into raw ndarrays (either by creating new ones or passing a view) or use the "subok = False" kwarg parameter to break a possible overloading loop.

The sequence of events is something like this: 1. You call np.sin(x) 2. The np.sin ufunc looks at x, sees the _numpy_ufunc_ attribute, and calls x._numpy_ufunc_(np.sin, x) 3. _numpy_ufunc_ uses np.sin.name (which is "sin") to get the correct my_sin function to call 4A. If my_sin called np.sin(x), we would go back to 1. and get an infinite loop 4B. If x is a subclass of ndarray, my_sin can call np.sin(x, subok=False), as this disables the subclass overloading mechanism. 4C. If x is not a subclass of ndarray, x needs to produce an ndarray, for instance it might have an x.arr property. Then it can call np.sin(x.arr)

Ok, that seems straightforward and, for what its worth, it looks like it would meet my needs. However, I wonder if the _numpy_func_ mechanism is the best approach. This is a bit sketchy, but why not do something like: class Proxy: def __init__(self, ufunc, *args): self._ufunc = ufunc self._args = args def __call__(self, func): self._ufunc._registry[tuple(type(arg) for arg in self._args)] = func return func class UfuncObject: ... def __call__(self, *args, **kwargs): func = self._registry.get(tuple(type(arg) for arg in args), None) if func is None: raise TypeError return func(*args, **kwargs) def register(self, *args): return Proxy(self, *args) @np.sin.register(Quantity) def sin(pq): if pq.units != degrees: pq = pq.rescale(degrees) temp = np.sin(pq.view(np.ndarray)) return Quantity(temp, copy=False) This way, classes don't have to implement special methods to support ufunc registration, special attributes to claim primacy in ufunc registration lookup, special versions of the functions for each numpy ufunc, *and* the logic to determine whether the combination of arguments is supported. By that I mean, if I call np.sum with a quantity and a masked array, and Quantity wins the __array_priority__ competition, then I also need to check that my special sum function(s) know how to operate on that combination of inputs. With the decorator approach, I just need to implement the special versions of the ufuncs, and the decorators handle the logic of knowing what combinations of arguments are supported. It might be worth considering using ABCs for registration and have UfuncObject use isinstance to determine the appropriate special function to call. Darren

What is the status of this proposal? On Wed, Jun 22, 2011 at 6:56 PM, Mark Wiebe wrote:

On Wed, Jun 22, 2011 at 4:57 PM, Darren Dale wrote:

...

On Wed, Jun 22, 2011 at 1:31 PM, Mark Wiebe wrote:

...

On Wed, Jun 22, 2011 at 7:34 AM, Lluís wrote:

...

Darren Dale writes:

...

On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe wrote:

...

On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris wrote: > How does the ufunc get called so it doesn't get caught in an > endless > loop?

...

[...]

...

...

The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.

...

I didn't understand, could you please expand on that or show an example?

As I understood the initial description and examples, the ufunc overload will keep being used as long as its arguments are of classes that declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" attribute).

Thus Mark's comment saying that you have to either transform the arguments into raw ndarrays (either by creating new ones or passing a view) or use the "subok = False" kwarg parameter to break a possible overloading loop.

The sequence of events is something like this: 1. You call np.sin(x) 2. The np.sin ufunc looks at x, sees the _numpy_ufunc_ attribute, and calls x._numpy_ufunc_(np.sin, x) 3. _numpy_ufunc_ uses np.sin.name (which is "sin") to get the correct my_sin function to call 4A. If my_sin called np.sin(x), we would go back to 1. and get an infinite loop 4B. If x is a subclass of ndarray, my_sin can call np.sin(x, subok=False), as this disables the subclass overloading mechanism. 4C. If x is not a subclass of ndarray, x needs to produce an ndarray, for instance it might have an x.arr property. Then it can call np.sin(x.arr)

Ok, that seems straightforward and, for what its worth, it looks like it would meet my needs. However, I wonder if the _numpy_func_ mechanism is the best approach. This is a bit sketchy, but why not do something like:

class Proxy:

   def __init__(self, ufunc, *args):        self._ufunc = ufunc        self._args = args

   def __call__(self, func):        self._ufunc._registry[tuple(type(arg) for arg in self._args)] = func        return func

class UfuncObject:

   ...

   def __call__(self, *args, **kwargs):        func = self._registry.get(tuple(type(arg) for arg in args), None)        if func is None:            raise TypeError        return func(*args, **kwargs)

   def register(self, *args):        return Proxy(self, *args)

@np.sin.register(Quantity) def sin(pq):    if pq.units != degrees:        pq = pq.rescale(degrees)    temp = np.sin(pq.view(np.ndarray))    return Quantity(temp, copy=False)

This way, classes don't have to implement special methods to support ufunc registration, special attributes to claim primacy in ufunc registration lookup, special versions of the functions for each numpy ufunc, *and* the logic to determine whether the combination of arguments is supported. By that I mean, if I call np.sum with a quantity and a masked array, and Quantity wins the __array_priority__ competition, then I also need to check that my special sum function(s) know how to operate on that combination of inputs. With the decorator approach, I just need to implement the special versions of the ufuncs, and the decorators handle the logic of knowing what combinations of arguments are supported.

It might be worth considering using ABCs for registration and have UfuncObject use isinstance to determine the appropriate special function to call.

The thing I'm not sure about with this idea is how to do something general to modify all ufuncs for a particular class in one swoop. Having to individually override everything would be a hassle, I think. I also don't think this approach saves very much effort compared to the _numpy_ufunc_ call approach, checking the types and raising NotImplemented if they aren't what's wanted is pretty much the only difference, and that check could still be handled by a decorator like you're showing. -Mark

...

Darren _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion