__numpy_ufunc__ and 1.9 release
hi, as you may know we want to release numpy 1.9 soon. We should have solved most indexing regressions the first beta showed.
The remaining blockers are finishing the new __numpy_ufunc__ feature. This feature should allow for alternative method to overriding the behavior of ufuncs from subclasses. It is described here: https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst
The current blocker issues are: https://github.com/numpy/numpy/issues/4753 https://github.com/numpy/numpy/pull/4815
I'm not to familiar with all the complications of subclassing so I can't really say how hard this is to solve. My issue is that it there still seems to be debate on how to handle operator overriding correctly and I am opposed to releasing a numpy with yet another experimental feature that may or may not be finished sometime later. Having datetime in infinite experimental state is bad enough. I think nobody is served well if we release 1.9 with the feature prematurely based on a not representative set of users and the later after more users showed up see we have to change its behavior.
So I'm wondering if we should delay the introduction of this feature to 1.10 or is it important enough to wait until there is a consensus on the remaining issues?
Hi,
15.07.2014 21:06, Julian Taylor kirjoitti: [clip: __numpy_ufunc__]
So I'm wondering if we should delay the introduction of this feature to 1.10 or is it important enough to wait until there is a consensus on the remaining issues?
My 10c:
The feature is not so much in hurry that it alone should delay 1.9. Moreover, it's best for everyone that it is bug-free on the first go, and it gets some real-world testing before the release. Better safe than sorry.
I'd pull it out from 1.9.x branch, and iron out the remaining wrinkles before 1.10.
Pauli
On Wed, Jul 16, 2014 at 11:16 AM, Pauli Virtanen pav@iki.fi wrote:
Hi,
15.07.2014 21:06, Julian Taylor kirjoitti: [clip: __numpy_ufunc__]
So I'm wondering if we should delay the introduction of this feature to 1.10 or is it important enough to wait until there is a consensus on the remaining issues?
My 10c:
The feature is not so much in hurry that it alone should delay 1.9. Moreover, it's best for everyone that it is bug-free on the first go, and it gets some real-world testing before the release. Better safe than sorry.
I'd pull it out from 1.9.x branch, and iron out the remaining wrinkles before 1.10.
Thanks Pauli, your opinion on the matter is what I needed to see and I'll take it as dispositive.
Chuck
On Tue, Jul 15, 2014 at 7:06 PM, Julian Taylor jtaylor.debian@googlemail.com wrote:
hi, as you may know we want to release numpy 1.9 soon. We should have solved most indexing regressions the first beta showed.
The remaining blockers are finishing the new __numpy_ufunc__ feature. This feature should allow for alternative method to overriding the behavior of ufuncs from subclasses. It is described here: https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst
The current blocker issues are: https://github.com/numpy/numpy/issues/4753 https://github.com/numpy/numpy/pull/4815
I'm not to familiar with all the complications of subclassing so I can't really say how hard this is to solve. My issue is that it there still seems to be debate on how to handle operator overriding correctly and I am opposed to releasing a numpy with yet another experimental feature that may or may not be finished sometime later. Having datetime in infinite experimental state is bad enough. I think nobody is served well if we release 1.9 with the feature prematurely based on a not representative set of users and the later after more users showed up see we have to change its behavior.
So I'm wondering if we should delay the introduction of this feature to 1.10 or is it important enough to wait until there is a consensus on the remaining issues?
-1 on delaying the release (but you knew I'd say that)
I don't have a strong feeling about whether or not we should disable __numpy_ufunc__ for the 1.9 release based on those bugs. They don't seem obviously catastrophic to me, but you make a good point about datetime. I think it's your call as release manager...
-n
On 15.07.2014 20:06, Julian Taylor wrote:
hi, as you may know we want to release numpy 1.9 soon. We should have solved most indexing regressions the first beta showed.
The remaining blockers are finishing the new __numpy_ufunc__ feature. This feature should allow for alternative method to overriding the behavior of ufuncs from subclasses. It is described here: https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst
The current blocker issues are: https://github.com/numpy/numpy/issues/4753 https://github.com/numpy/numpy/pull/4815
I'm not to familiar with all the complications of subclassing so I can't really say how hard this is to solve. My issue is that it there still seems to be debate on how to handle operator overriding correctly and I am opposed to releasing a numpy with yet another experimental feature that may or may not be finished sometime later. Having datetime in infinite experimental state is bad enough. I think nobody is served well if we release 1.9 with the feature prematurely based on a not representative set of users and the later after more users showed up see we have to change its behavior.
So I'm wondering if we should delay the introduction of this feature to 1.10 or is it important enough to wait until there is a consensus on the remaining issues?
So its been a week and we got a few answers and new issues. To summarize: - to my knowledge no progress was made on the issues - scipy already has a released version using the current implementation - no very loud objections to delaying the feature to 1.10 - I am still unfamiliar with the problematics of subclassing, but don't want to release something new which has unsolved issues.
That scipy already uses it in a released version (0.14) is very problematic. Can maybe someone give some insight if the potential changes to resolve the remaining issues would break scipy? If so we have following choices: - declare what we have as final and close the remaining issues as 'won't fix'. Any changes would have to have a new name __numpy_ufunc2__ or a somehow versioned the interface - delay the introduction, potentially breaking scipy 0.14 when numpy 1.10 is released.
I would like to get the next (and last) numpy 1.9 beta out soon, so I would propose to make a decision until this Saturday the 26.02.2014 however misinformed it may be.
Please note that the numpy 1.10 release cycle is likely going to be a very long one as we are currently planning to change a bunch of default behaviours that currently raise deprecation warnings and possibly will try to fix string types, text IO and datetime. Please see the future changes notes in the current 1.9.x release notes. If we delay numpy_ufunc it is not unlikely that it will take a year until we release 1.10. Though we could still put it into a earlier 1.9.1.
Cheers, Julian
23.07.2014, 20:37, Julian Taylor kirjoitti: [clip: __numpy_ufunc__]
So its been a week and we got a few answers and new issues. To summarize: - to my knowledge no progress was made on the issues - scipy already has a released version using the current implementation - no very loud objections to delaying the feature to 1.10 - I am still unfamiliar with the problematics of subclassing, but don't want to release something new which has unsolved issues.
That scipy already uses it in a released version (0.14) is very problematic. Can maybe someone give some insight if the potential changes to resolve the remaining issues would break scipy?
If so we have following choices:
- declare what we have as final and close the remaining issues as
'won't fix'. Any changes would have to have a new name __numpy_ufunc2__ or a somehow versioned the interface - delay the introduction, potentially breaking scipy 0.14 when numpy 1.10 is released.
I would like to get the next (and last) numpy 1.9 beta out soon, so I would propose to make a decision until this Saturday the 26.02.2014 however misinformed it may be.
It seems fairly unlikely to me that the `__numpy_ufunc__` interface itself requires any changes. I believe the definition of the interface is quite safe to consider as fixed --- it is a fairly straighforward hook for Numpy ufuncs. (There are also no essential changes in it since last year.)
For the binary operator overriding, Scipy sets the constraint that
ndarray * spmatrix
MUST call spmatrix.__rmul__ even if spmatrix.__numpy_ufunc__ is defined. spmatrixes are not ndarray subclasses, and various subclassing problems do not enter here.
Note that this binop discussion is somewhat separate from the __numpy_ufunc__ interface itself. The only information available about it at the binop stage is `hasattr(other, '__numpy_ufunc__')`.
***
Regarding the blockers:
(1) https://github.com/numpy/numpy/issues/4753
This is a bug in the argument normalization --- output arguments are not checked for the presence of "__numpy_ufunc__" if they are passed as keyword arguments (as a positional argument it works). It's a bug in the implementation, but I don't think it is really a blocker.
Scipy sparse matrices will in practice seldom be used as output args for ufuncs.
***
(2) https://github.com/numpy/numpy/pull/4815
The is open question concerns semantics of `__numpy_ufunc__` versus Python operator overrides. When should ndarray.__mul__(other) return NotImplemented?
Scipy sparse matrices are not subclasses of ndarray, so the code in question in Numpy gets to run only for
ndarray * spmatrix
This provides a constraint to what solution we can choose in Numpy to deal with the issue:
ndarray.__mul__(spmatrix) MUST continue to return NotImplemented
This is the current behavior, and cannot be changed: it is not possible to defer this to __numpy_ufunc__(ufunc=np.multiply), because sparse matrices define `*` as the matrix multiply, and not the elementwise multiply. (This settles one line of discussion in the issues --- ndarray should defer.)
How Numpy currently determines whether to return NotImplemented in this case or to call np.multiply(self, other) is by comparing `__array_priority__` attributes of `self` and `other`. Scipy sparse matrices define an `__array_priority__` larger than ndarrays, which then makes a NotImplemented be returned.
The idea in the __numpy_ufunc__ NEP was to replace this with `hasattr(other, '__numpy_ufunc__') and hasattr(other, '__rmul__')`. However, when both self and other are ndarray subclasses in a certain configuration, both end up returning NotImplemented, and Python raises TypeError.
The `__array_priority__` mechanism is also broken in some of the subclassing cases: https://github.com/numpy/numpy/issues/4766
As far as I see, the backward compatibility requirement from Scipy only rules out the option that ndarray.__mul__(other) should unconditionally call `np.add(self, other)`.
We have some freedom how to solve the binop vs. subclass issues. It's possible to e.g. retain the __array_priority__ stuff as a backward compatibility measure as we do currently.
participants (4)
-
Charles R Harris -
Julian Taylor -
Nathaniel Smith -
Pauli Virtanen