replacing the mechanism for dispatching ufuncs
NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.
Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like.
The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default.
The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.
Thanks, Mark
A simple class which overrides the ufuncs might look as follows:
def sin(ufunc, *args, **kwargs): # Convert degrees to radians args[0] = np.deg2rad(args[0]) # Return a regular array, since the result is not in degrees return ufunc(*args, **kwargs)
class MyDegreesClass: """Array-like object with a degrees unit"""
def __init__(arr): self.arr = arr
def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: raise TypeError, 'ufunc %s incompatible with MyDegreesClass' % ufunc.name
A more complex example will be something like this:
def my_general_ufunc(ufunc, *args, **kwargs): # Extract the 'out' argument. This only supports ufuncs with # one output, currently. out = kwargs.get('out') if len(args) > ufunc.nin: if out is None: out = args[ufunc.nin] else: raise ValueError, "'out' given as both a position and keyword argument"
# Just want the inputs from here on args = args[:ufunc.nin]
# Strip out MyArrayClass, but allow operations with regular ndarrays raw_in = [] for a in args: if isinstance(a, MyArrayClass): raw_in.append(a.arr) else: raw_in.append(a)
# Allocate the output array if not out is None: if isinstance(out, MyArrayClass): raise TypeError, "'out' must have type MyArrayClass" else: # Create the output array, obeying the 'order' parameter, # but disallowing subclasses out = np.broadcast_empty_like([args, order=kwargs.get('order'), dtype=ufunc.result_type(args), subok=False)
# Override the output argument kwargs['out'] = out.arr
# Call the ufunc ufunc(*args, **kwargs)
# Return the output return out
class MyArrayClass: def __init__(arr): self.arr = arr
def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: return my_general_ufunc(ufunc, *args, **kwargs)
On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe mwwiebe@gmail.com wrote:
NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.
The current approach is at a dead end, so something better needs to be done.
Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like.
The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default.
The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.
How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes that don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.
Thanks, Mark
A simple class which overrides the ufuncs might look as follows:
def sin(ufunc, *args, **kwargs): # Convert degrees to radians args[0] = np.deg2rad(args[0]) # Return a regular array, since the result is not in degrees return ufunc(*args, **kwargs)
class MyDegreesClass: """Array-like object with a degrees unit"""
def __init__(arr): self.arr = arr def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: raise TypeError, 'ufunc %s incompatible with MyDegreesClass' %ufunc.name
A more complex example will be something like this:
def my_general_ufunc(ufunc, *args, **kwargs): # Extract the 'out' argument. This only supports ufuncs with # one output, currently. out = kwargs.get('out') if len(args) > ufunc.nin: if out is None: out = args[ufunc.nin] else: raise ValueError, "'out' given as both a position and keyword argument"
# Just want the inputs from here on args = args[:ufunc.nin] # Strip out MyArrayClass, but allow operations with regular ndarrays raw_in = [] for a in args: if isinstance(a, MyArrayClass): raw_in.append(a.arr) else: raw_in.append(a) # Allocate the output array if not out is None: if isinstance(out, MyArrayClass): raise TypeError, "'out' must have type MyArrayClass" else: # Create the output array, obeying the 'order' parameter, # but disallowing subclasses out = np.broadcast_empty_like([args, order=kwargs.get('order'), dtype=ufunc.result_type(args), subok=False) # Override the output argument kwargs['out'] = out.arr # Call the ufunc ufunc(*args, **kwargs) # Return the output return outclass MyArrayClass: def __init__(arr): self.arr = arr
def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: return my_general_ufunc(ufunc, *args, **kwargs)
Chuck
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe mwwiebe@gmail.com wrote:
NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.
The current approach is at a dead end, so something better needs to be done.
Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like.
The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default.
The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.
How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes that don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs. Supporting objects that aren't ndarray subclasses is one of the purposes for this approach, and neither of my two example cases subclassed ndarray.
-Mark
Thanks, Mark
A simple class which overrides the ufuncs might look as follows:
def sin(ufunc, *args, **kwargs): # Convert degrees to radians args[0] = np.deg2rad(args[0]) # Return a regular array, since the result is not in degrees return ufunc(*args, **kwargs)
class MyDegreesClass: """Array-like object with a degrees unit"""
def __init__(arr): self.arr = arr def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: raise TypeError, 'ufunc %s incompatible with MyDegreesClass' %ufunc.name
A more complex example will be something like this:
def my_general_ufunc(ufunc, *args, **kwargs): # Extract the 'out' argument. This only supports ufuncs with # one output, currently. out = kwargs.get('out') if len(args) > ufunc.nin: if out is None: out = args[ufunc.nin] else: raise ValueError, "'out' given as both a position and keyword argument"
# Just want the inputs from here on args = args[:ufunc.nin] # Strip out MyArrayClass, but allow operations with regular ndarrays raw_in = [] for a in args: if isinstance(a, MyArrayClass): raw_in.append(a.arr) else: raw_in.append(a) # Allocate the output array if not out is None: if isinstance(out, MyArrayClass): raise TypeError, "'out' must have type MyArrayClass" else: # Create the output array, obeying the 'order' parameter, # but disallowing subclasses out = np.broadcast_empty_like([args, order=kwargs.get('order'), dtype=ufunc.result_type(args), subok=False) # Override the output argument kwargs['out'] = out.arr # Call the ufunc ufunc(*args, **kwargs) # Return the output return outclass MyArrayClass: def __init__(arr): self.arr = arr
def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: return my_general_ufunc(ufunc, *args, **kwargs)Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe mwwiebe@gmail.com wrote:
NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.
The current approach is at a dead end, so something better needs to be done.
Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like.
The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default.
The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.
How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes that don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs. Supporting objects that aren't ndarray subclasses is one of the purposes for this approach, and neither of my two example cases subclassed ndarray.
Sounds good. Many of the current uses of __array_wrap__ that I am aware of are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled?
<snip>
Chuck
On Tue, Jun 21, 2011 at 2:28 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe mwwiebe@gmail.com wrote:
NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.
The current approach is at a dead end, so something better needs to be done.
Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like. The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default. The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.
How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes that don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs. Supporting objects that aren't ndarray subclasses is one of the purposes for this approach, and neither of my two example cases subclassed ndarray.
Sounds good. Many of the current uses of __array_wrap__ that I am aware of are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled?
I contributed the __array_prepare__ method a while back so classes could raise errors before the array data is modified in place. Specifically, I was concerned about units support in my quantities package (http://pypi.python.org/pypi/quantities). But I agree that this approach is needs to be reconsidered. It would be nice for subclasses to have an opportunity to intercept and process the values passed to a ufunc on their way in. For example, it would be nice if when I did np.cos(1.5 degrees), my subclass could intercept the value and pass a new one on to the ufunc machinery that is expressed in radians. I thought PJ Eby's generic functions PEP would be a really good way to handle ufuncs, but the PEP has stagnated.
Darren
On Tue, Jun 21, 2011 at 12:46 PM, Darren Dale dsdale24@gmail.com wrote:
On Tue, Jun 21, 2011 at 2:28 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe mwwiebe@gmail.com
wrote:
NumPy has a mechanism built in to allow subclasses to adjust or
override
aspects of the ufunc behavior. While this goal is important, this
mechanism
only allows for very limited customization, making for instance the
masked
arrays unable to work with the native ufuncs in a full and proper way.
I
would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I
will
describe below. If you've ever used these mechanisms, please review
this
design to see if it meets your needs.
The current approach is at a dead end, so something better needs to be done.
Any class type which would like to override its behavior in ufuncs
would
define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the
priority
battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function,
broadcast_empty_like.
The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to
the
ufunc, which may follow a different convention than regular arithmetic
type
promotion. This allows for a class to create an output array of the
correct
type to pass to the ufunc if it needs to be different than the
default.
The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for
producing the
output, instead of just one.
How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes
that
don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.
The function being called needs to ensure this, either by extracting a
raw
ndarray from instances of its class, or adding a 'subok = False'
parameter
to the kwargs. Supporting objects that aren't ndarray subclasses is one
of
the purposes for this approach, and neither of my two example cases subclassed ndarray.
Sounds good. Many of the current uses of __array_wrap__ that I am aware
of
are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled?
I contributed the __array_prepare__ method a while back so classes could raise errors before the array data is modified in place. Specifically, I was concerned about units support in my quantities package (http://pypi.python.org/pypi/quantities). But I agree that this approach is needs to be reconsidered. It would be nice for subclasses to have an opportunity to intercept and process the values passed to a ufunc on their way in. For example, it would be nice if when I did np.cos(1.5 degrees), my subclass could intercept the value and pass a new one on to the ufunc machinery that is expressed in radians. I thought PJ Eby's generic functions PEP would be a really
Link to PEP-3124 http://tinyurl.com/3brnk6.
Chuck
On Tue, Jun 21, 2011 at 1:46 PM, Darren Dale dsdale24@gmail.com wrote:
On Tue, Jun 21, 2011 at 2:28 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe mwwiebe@gmail.com
wrote:
NumPy has a mechanism built in to allow subclasses to adjust or
override
aspects of the ufunc behavior. While this goal is important, this
mechanism
only allows for very limited customization, making for instance the
masked
arrays unable to work with the native ufuncs in a full and proper way.
I
would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I
will
describe below. If you've ever used these mechanisms, please review
this
design to see if it meets your needs.
The current approach is at a dead end, so something better needs to be done.
Any class type which would like to override its behavior in ufuncs
would
define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the
priority
battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function,
broadcast_empty_like.
The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to
the
ufunc, which may follow a different convention than regular arithmetic
type
promotion. This allows for a class to create an output array of the
correct
type to pass to the ufunc if it needs to be different than the
default.
The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for
producing the
output, instead of just one.
How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes
that
don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.
The function being called needs to ensure this, either by extracting a
raw
ndarray from instances of its class, or adding a 'subok = False'
parameter
to the kwargs. Supporting objects that aren't ndarray subclasses is one
of
the purposes for this approach, and neither of my two example cases subclassed ndarray.
Sounds good. Many of the current uses of __array_wrap__ that I am aware
of
are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled?
I contributed the __array_prepare__ method a while back so classes could raise errors before the array data is modified in place. Specifically, I was concerned about units support in my quantities package (http://pypi.python.org/pypi/quantities). But I agree that this approach is needs to be reconsidered. It would be nice for subclasses to have an opportunity to intercept and process the values passed to a ufunc on their way in. For example, it would be nice if when I did np.cos(1.5 degrees), my subclass could intercept the value and pass a new one on to the ufunc machinery that is expressed in radians. I thought PJ Eby's generic functions PEP would be a really good way to handle ufuncs, but the PEP has stagnated.
I made one of my examples overriding sin with degrees, because I think this overloading method can work well for a physical quantities library.
-Mark
Darren _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Jun 21, 2011 at 1:28 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Tue, Jun 21, 2011 at 11:57 AM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Mon, Jun 20, 2011 at 12:32 PM, Mark Wiebe mwwiebe@gmail.com wrote:
NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs.
The current approach is at a dead end, so something better needs to be done.
Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows:
return arr._numpy_ufunc_(current_ufunc, *args, **kwargs)
To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like.
The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default.
The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one.
How does the ufunc get called so it doesn't get caught in an endless loop? I like the proposed method if it can also be used for classes that don't subclass ndarray. Masked array, for instance, should probably not subclass ndarray.
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs. Supporting objects that aren't ndarray subclasses is one of the purposes for this approach, and neither of my two example cases subclassed ndarray.
Sounds good. Many of the current uses of __array_wrap__ that I am aware of are in the wrappers in the linalg module and don't go through the ufunc machinery. How would that be handled?
Those could stay as they are, and just the ufunc usage of __array_wrap__ can be deprecated. For classes which currently use __array_wrap__, they would just need to also implement _numpy_ufunc_ to eliminate any deprecation messages.
-Mark
<snip>
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Tue, 21 Jun 2011 16:43:13 -0500, Mark Wiebe wrote: [clip: __array_wrap__]
Those could stay as they are, and just the ufunc usage of __array_wrap__ can be deprecated. For classes which currently use __array_wrap__, they would just need to also implement _numpy_ufunc_ to eliminate any deprecation messages.
Do you mean that the new mechanism would not be able to do the same thing here?
Preservation of array subclasses in linalg functions is not very uniform, and will likely need fixes in several of the functions. Since new code in any case would need to be written, I'd prefer using the "new" approach and so leaving us the option of marking the "old" approach deprecated.
Pauli
On Wed, Jun 22, 2011 at 5:08 AM, Pauli Virtanen pav@iki.fi wrote:
Tue, 21 Jun 2011 16:43:13 -0500, Mark Wiebe wrote: [clip: __array_wrap__]
Those could stay as they are, and just the ufunc usage of __array_wrap__ can be deprecated. For classes which currently use __array_wrap__, they would just need to also implement _numpy_ufunc_ to eliminate any deprecation messages.
Do you mean that the new mechanism would not be able to do the same thing here?
It would have to be generalized a bit more to support these usages, because some functions produce outputs with different shapes, and the inputs may not be broadcast together in the same manner as in the element-wise ufuncs.
Preservation of array subclasses in linalg functions is not very uniform, and will likely need fixes in several of the functions. Since new code in any case would need to be written, I'd prefer using the "new" approach and so leaving us the option of marking the "old" approach deprecated.
I think creating a @ufunc_overload(...) decorator for specifying the ufunc properties like nin and nout might be a nice way to generalize it.
-Mark
Pauli
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
How does the ufunc get called so it doesn't get caught in an endless loop?
[...]
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.
I didn't understand, could you please expand on that or show an example?
Darren
Darren Dale writes:
On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
How does the ufunc get called so it doesn't get caught in an endless loop?
[...]
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.
I didn't understand, could you please expand on that or show an example?
As I understood the initial description and examples, the ufunc overload will keep being used as long as its arguments are of classes that declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" attribute).
Thus Mark's comment saying that you have to either transform the arguments into raw ndarrays (either by creating new ones or passing a view) or use the "subok = False" kwarg parameter to break a possible overloading loop.
Lluis
On Wed, Jun 22, 2011 at 7:34 AM, Lluís xscript@gmx.net wrote:
Darren Dale writes:
On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
How does the ufunc get called so it doesn't get caught in an endless
loop?
[...]
The function being called needs to ensure this, either by extracting a
raw
ndarray from instances of its class, or adding a 'subok = False'
parameter
to the kwargs.
I didn't understand, could you please expand on that or show an example?
As I understood the initial description and examples, the ufunc overload will keep being used as long as its arguments are of classes that declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" attribute).
Thus Mark's comment saying that you have to either transform the arguments into raw ndarrays (either by creating new ones or passing a view) or use the "subok = False" kwarg parameter to break a possible overloading loop.
The sequence of events is something like this:
1. You call np.sin(x) 2. The np.sin ufunc looks at x, sees the _numpy_ufunc_ attribute, and calls x._numpy_ufunc_(np.sin, x) 3. _numpy_ufunc_ uses np.sin.name (which is "sin") to get the correct my_sin function to call 4A. If my_sin called np.sin(x), we would go back to 1. and get an infinite loop 4B. If x is a subclass of ndarray, my_sin can call np.sin(x, subok=False), as this disables the subclass overloading mechanism. 4C. If x is not a subclass of ndarray, x needs to produce an ndarray, for instance it might have an x.arr property. Then it can call np.sin(x.arr)
-Mark
Lluis
-- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Wed, Jun 22, 2011 at 1:31 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Wed, Jun 22, 2011 at 7:34 AM, Lluís xscript@gmx.net wrote:
Darren Dale writes:
On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
How does the ufunc get called so it doesn't get caught in an endless loop?
[...]
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.
I didn't understand, could you please expand on that or show an example?
As I understood the initial description and examples, the ufunc overload will keep being used as long as its arguments are of classes that declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" attribute).
Thus Mark's comment saying that you have to either transform the arguments into raw ndarrays (either by creating new ones or passing a view) or use the "subok = False" kwarg parameter to break a possible overloading loop.
The sequence of events is something like this:
- You call np.sin(x)
- The np.sin ufunc looks at x, sees the _numpy_ufunc_ attribute, and calls
x._numpy_ufunc_(np.sin, x) 3. _numpy_ufunc_ uses np.sin.name (which is "sin") to get the correct my_sin function to call 4A. If my_sin called np.sin(x), we would go back to 1. and get an infinite loop 4B. If x is a subclass of ndarray, my_sin can call np.sin(x, subok=False), as this disables the subclass overloading mechanism. 4C. If x is not a subclass of ndarray, x needs to produce an ndarray, for instance it might have an x.arr property. Then it can call np.sin(x.arr)
Ok, that seems straightforward and, for what its worth, it looks like it would meet my needs. However, I wonder if the _numpy_func_ mechanism is the best approach. This is a bit sketchy, but why not do something like:
class Proxy:
def __init__(self, ufunc, *args): self._ufunc = ufunc self._args = args
def __call__(self, func): self._ufunc._registry[tuple(type(arg) for arg in self._args)] = func return func
class UfuncObject:
...
def __call__(self, *args, **kwargs): func = self._registry.get(tuple(type(arg) for arg in args), None) if func is None: raise TypeError return func(*args, **kwargs)
def register(self, *args): return Proxy(self, *args)
@np.sin.register(Quantity) def sin(pq): if pq.units != degrees: pq = pq.rescale(degrees) temp = np.sin(pq.view(np.ndarray)) return Quantity(temp, copy=False)
This way, classes don't have to implement special methods to support ufunc registration, special attributes to claim primacy in ufunc registration lookup, special versions of the functions for each numpy ufunc, *and* the logic to determine whether the combination of arguments is supported. By that I mean, if I call np.sum with a quantity and a masked array, and Quantity wins the __array_priority__ competition, then I also need to check that my special sum function(s) know how to operate on that combination of inputs. With the decorator approach, I just need to implement the special versions of the ufuncs, and the decorators handle the logic of knowing what combinations of arguments are supported.
It might be worth considering using ABCs for registration and have UfuncObject use isinstance to determine the appropriate special function to call.
Darren
On Wed, Jun 22, 2011 at 4:57 PM, Darren Dale dsdale24@gmail.com wrote:
On Wed, Jun 22, 2011 at 1:31 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Wed, Jun 22, 2011 at 7:34 AM, Lluís xscript@gmx.net wrote:
Darren Dale writes:
On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe mwwiebe@gmail.com
wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote:
How does the ufunc get called so it doesn't get caught in an endless loop?
[...]
The function being called needs to ensure this, either by extracting
a
raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.
I didn't understand, could you please expand on that or show an
example?
As I understood the initial description and examples, the ufunc overload will keep being used as long as its arguments are of classes that declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" attribute).
Thus Mark's comment saying that you have to either transform the arguments into raw ndarrays (either by creating new ones or passing a view) or use the "subok = False" kwarg parameter to break a possible overloading loop.
The sequence of events is something like this:
- You call np.sin(x)
- The np.sin ufunc looks at x, sees the _numpy_ufunc_ attribute, and
calls
x._numpy_ufunc_(np.sin, x) 3. _numpy_ufunc_ uses np.sin.name (which is "sin") to get the correct
my_sin
function to call 4A. If my_sin called np.sin(x), we would go back to 1. and get an
infinite
loop 4B. If x is a subclass of ndarray, my_sin can call np.sin(x,
subok=False),
as this disables the subclass overloading mechanism. 4C. If x is not a subclass of ndarray, x needs to produce an ndarray, for instance it might have an x.arr property. Then it can call np.sin(x.arr)
Ok, that seems straightforward and, for what its worth, it looks like it would meet my needs. However, I wonder if the _numpy_func_ mechanism is the best approach. This is a bit sketchy, but why not do something like:
class Proxy:
def __init__(self, ufunc, *args): self._ufunc = ufunc self._args = args
def __call__(self, func): self._ufunc._registry[tuple(type(arg) for arg in self._args)] = func return func
class UfuncObject:
...
def __call__(self, *args, **kwargs): func = self._registry.get(tuple(type(arg) for arg in args), None) if func is None: raise TypeError return func(*args, **kwargs)
def register(self, *args): return Proxy(self, *args)
@np.sin.register(Quantity) def sin(pq): if pq.units != degrees: pq = pq.rescale(degrees) temp = np.sin(pq.view(np.ndarray)) return Quantity(temp, copy=False)
This way, classes don't have to implement special methods to support ufunc registration, special attributes to claim primacy in ufunc registration lookup, special versions of the functions for each numpy ufunc, *and* the logic to determine whether the combination of arguments is supported. By that I mean, if I call np.sum with a quantity and a masked array, and Quantity wins the __array_priority__ competition, then I also need to check that my special sum function(s) know how to operate on that combination of inputs. With the decorator approach, I just need to implement the special versions of the ufuncs, and the decorators handle the logic of knowing what combinations of arguments are supported.
It might be worth considering using ABCs for registration and have UfuncObject use isinstance to determine the appropriate special function to call.
The thing I'm not sure about with this idea is how to do something general to modify all ufuncs for a particular class in one swoop. Having to individually override everything would be a hassle, I think. I also don't think this approach saves very much effort compared to the _numpy_ufunc_ call approach, checking the types and raising NotImplemented if they aren't what's wanted is pretty much the only difference, and that check could still be handled by a decorator like you're showing.
-Mark
Darren _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
What is the status of this proposal?
On Wed, Jun 22, 2011 at 6:56 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Wed, Jun 22, 2011 at 4:57 PM, Darren Dale dsdale24@gmail.com wrote:
On Wed, Jun 22, 2011 at 1:31 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Wed, Jun 22, 2011 at 7:34 AM, Lluís xscript@gmx.net wrote:
Darren Dale writes:
On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe mwwiebe@gmail.com wrote:
On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris charlesr.harris@gmail.com wrote: > How does the ufunc get called so it doesn't get caught in an > endless > loop?
[...]
The function being called needs to ensure this, either by extracting a raw ndarray from instances of its class, or adding a 'subok = False' parameter to the kwargs.
I didn't understand, could you please expand on that or show an example?
As I understood the initial description and examples, the ufunc overload will keep being used as long as its arguments are of classes that declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" attribute).
Thus Mark's comment saying that you have to either transform the arguments into raw ndarrays (either by creating new ones or passing a view) or use the "subok = False" kwarg parameter to break a possible overloading loop.
The sequence of events is something like this:
- You call np.sin(x)
- The np.sin ufunc looks at x, sees the _numpy_ufunc_ attribute, and
calls x._numpy_ufunc_(np.sin, x) 3. _numpy_ufunc_ uses np.sin.name (which is "sin") to get the correct my_sin function to call 4A. If my_sin called np.sin(x), we would go back to 1. and get an infinite loop 4B. If x is a subclass of ndarray, my_sin can call np.sin(x, subok=False), as this disables the subclass overloading mechanism. 4C. If x is not a subclass of ndarray, x needs to produce an ndarray, for instance it might have an x.arr property. Then it can call np.sin(x.arr)
Ok, that seems straightforward and, for what its worth, it looks like it would meet my needs. However, I wonder if the _numpy_func_ mechanism is the best approach. This is a bit sketchy, but why not do something like:
class Proxy:
def __init__(self, ufunc, *args): self._ufunc = ufunc self._args = args
def __call__(self, func): self._ufunc._registry[tuple(type(arg) for arg in self._args)] = func return func
class UfuncObject:
...
def __call__(self, *args, **kwargs): func = self._registry.get(tuple(type(arg) for arg in args), None) if func is None: raise TypeError return func(*args, **kwargs)
def register(self, *args): return Proxy(self, *args)
@np.sin.register(Quantity) def sin(pq): if pq.units != degrees: pq = pq.rescale(degrees) temp = np.sin(pq.view(np.ndarray)) return Quantity(temp, copy=False)
This way, classes don't have to implement special methods to support ufunc registration, special attributes to claim primacy in ufunc registration lookup, special versions of the functions for each numpy ufunc, *and* the logic to determine whether the combination of arguments is supported. By that I mean, if I call np.sum with a quantity and a masked array, and Quantity wins the __array_priority__ competition, then I also need to check that my special sum function(s) know how to operate on that combination of inputs. With the decorator approach, I just need to implement the special versions of the ufuncs, and the decorators handle the logic of knowing what combinations of arguments are supported.
It might be worth considering using ABCs for registration and have UfuncObject use isinstance to determine the appropriate special function to call.
The thing I'm not sure about with this idea is how to do something general to modify all ufuncs for a particular class in one swoop. Having to individually override everything would be a hassle, I think. I also don't think this approach saves very much effort compared to the _numpy_ufunc_ call approach, checking the types and raising NotImplemented if they aren't what's wanted is pretty much the only difference, and that check could still be handled by a decorator like you're showing. -Mark
Darren _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (5)
-
Charles R Harris -
Darren Dale -
Lluís -
Mark Wiebe -
Pauli Virtanen