Hi All, Time to begin looking forward to the NumPy 1.16.x branch. I think there are three main topics to address: 1. current PRs that need review and merging, 2. critical fixes that need to be made, 3. status of `__array_function__`. The last probably needs some discussion. `__array_fuction__` seems to be working at this point, but does cause noticeable slowdowns in some function calls. I don't know if those slowdowns are significant in practice, the only way to discover that may be to make the release, or at least thorough testing of the release candidates, but we should at least discuss them and possible workarounds if needed. Trying to have things in good shape is important because 1.16.x will be the last release that supports Python 2.7, and even though we will be maintaining it for the next year, that will be easier if it is stable. As to the first two topics, I think we should try to be conservative at this point and look mostly for bug fixes and documentation updates. Thoughts? Chuck
Hi Chuck, For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in performance, this may be the best option. All the best, Marten
On Sun, Nov 4, 2018 at 10:32 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Hi Chuck,
For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in performance, this may be the best option. All the best,
Marten
I am also leaning towards this right now, depending on how long we plan to wait for releasing 1.16. It will take us at least a little while to sort out performance issues for __array_function__, I'd guess at least a few weeks. Then a blocker still might turn up during the release candidate process (though I think we've found most of the major bugs / downstream issues already through tests on NumPy's dev branch). Overall, it does feels a little misguided to rush in a change as pervasive as __array_function__ for a long term support release. If we exclude __array_function__ I expect the whole release process for 1.16 would go much smoother. We might even try to get 1.17 out faster than usual, so we can minimize the number of additional changes besides __array_function__ and going Python 3 only -- that's already a good bit of change. Note that if we make this change (reverting __array_function__), we'll need to revisit where we put a few deprecation warnings -- these will need to be restored into function bodies, not their dispatcher functions. Also: it would be really nice if we get matmul-as-ufunc in before (or at the same time) as __array_function__, so we have a complete story about it being possible to override everything in NumPy. This is another argument for delaying __array_function__, if matmul-as-ufunc can't make it in time for 1.16. Best, Stephan
On Sun, Nov 4, 2018 at 6:16 PM Stephan Hoyer <shoyer@gmail.com> wrote:
On Sun, Nov 4, 2018 at 10:32 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Hi Chuck,
For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in performance, this may be the best option. All the best,
Marten
I am also leaning towards this right now, depending on how long we plan to wait for releasing 1.16. It will take us at least a little while to sort out performance issues for __array_function__, I'd guess at least a few weeks. Then a blocker still might turn up during the release candidate process (though I think we've found most of the major bugs / downstream issues already through tests on NumPy's dev branch).
My tentative schedule is to branch in about two weeks, then allow 2 weeks of testing for rc1, possibly another two weeks for rc2, and then a final. so possibly about six weeks to final release. That leaves 2 to 4 weeks of slack before 2019.
Overall, it does feels a little misguided to rush in a change as pervasive as __array_function__ for a long term support release. If we exclude __array_function__ I expect the whole release process for 1.16 would go much smoother. We might even try to get 1.17 out faster than usual, so we can minimize the number of additional changes besides __array_function__ and going Python 3 only -- that's already a good bit of change.
I would like to get 1.17 out a bit early. I'm not sure how many backwards incompatible changes we want to have in the first post python2 release. My initial thoughts are to drop Python 2.7 testing, go to C99, and get the new fft in. Beyond that, I'm hesitant to start tearing out all the Python2 special casing in the first new release, although that could certainly be the main task for 1.17 and would clean up the code considerably. It might also be a good time to catch up on changing deprecations to errors. Thoughts on how to proceed are welcome.
Note that if we make this change (reverting __array_function__), we'll need to revisit where we put a few deprecation warnings -- these will need to be restored into function bodies, not their dispatcher functions.
Also: it would be really nice if we get matmul-as-ufunc in before (or at the same time) as __array_function__, so we have a complete story about it being possible to override everything in NumPy. This is another argument for delaying __array_function__, if matmul-as-ufunc can't make it in time for 1.16.
That's two votes for matmul-as-ufunc. How much would it cost to simply make __array_function__ a nop? Chuck
Thoughts on how to proceed are welcome.
I've been involved in scikit-image and that project tore out the python2 only code rather quickly after 2.7 support was dropped. I think it caused a few hiccups when backporting bugfixes. I imagine that `1.16.1` and `1.16.2` releases will come out quickly and as such, I think removing `if else` statements for python2 immediately after `1.16` is released will cause annoyances in the first few months bugs are being ironed out. My 2cents. On Sun, Nov 4, 2018 at 10:04 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Sun, Nov 4, 2018 at 6:16 PM Stephan Hoyer <shoyer@gmail.com> wrote:
On Sun, Nov 4, 2018 at 10:32 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Hi Chuck,
For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in performance, this may be the best option. All the best,
Marten
I am also leaning towards this right now, depending on how long we plan to wait for releasing 1.16. It will take us at least a little while to sort out performance issues for __array_function__, I'd guess at least a few weeks. Then a blocker still might turn up during the release candidate process (though I think we've found most of the major bugs / downstream issues already through tests on NumPy's dev branch).
My tentative schedule is to branch in about two weeks, then allow 2 weeks of testing for rc1, possibly another two weeks for rc2, and then a final. so possibly about six weeks to final release. That leaves 2 to 4 weeks of slack before 2019.
Overall, it does feels a little misguided to rush in a change as pervasive as __array_function__ for a long term support release. If we exclude __array_function__ I expect the whole release process for 1.16 would go much smoother. We might even try to get 1.17 out faster than usual, so we can minimize the number of additional changes besides __array_function__ and going Python 3 only -- that's already a good bit of change.
I would like to get 1.17 out a bit early. I'm not sure how many backwards incompatible changes we want to have in the first post python2 release. My initial thoughts are to drop Python 2.7 testing, go to C99, and get the new fft in. Beyond that, I'm hesitant to start tearing out all the Python2 special casing in the first new release, although that could certainly be the main task for 1.17 and would clean up the code considerably. It might also be a good time to catch up on changing deprecations to errors. Thoughts on how to proceed are welcome.
Note that if we make this change (reverting __array_function__), we'll need to revisit where we put a few deprecation warnings -- these will need to be restored into function bodies, not their dispatcher functions.
Also: it would be really nice if we get matmul-as-ufunc in before (or at the same time) as __array_function__, so we have a complete story about it being possible to override everything in NumPy. This is another argument for delaying __array_function__, if matmul-as-ufunc can't make it in time for 1.16.
That's two votes for matmul-as-ufunc. How much would it cost to simply make __array_function__ a nop?
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
For astropy, we also waiting a little before having a rip-python2-out fiesta. I think it is worth trying to get matmul in 1.16, independently of __array_function__ - it really belongs to ufunc overwrites and all the groundwork has been done. For __array_function__, is it at all an option to go to the disable-by-default step? I.e., by default have array_function_dispatch just return the implementation instead of wrapping it? Though perhaps reversion is indeed cleaner; most people who would like to play with it are quite able to install the development version... -- Marten On Sun, Nov 4, 2018 at 10:43 PM Mark Harfouche <mark.harfouche@gmail.com> wrote:
Thoughts on how to proceed are welcome.
I've been involved in scikit-image and that project tore out the python2 only code rather quickly after 2.7 support was dropped. I think it caused a few hiccups when backporting bugfixes. I imagine that `1.16.1` and `1.16.2` releases will come out quickly and as such, I think removing `if else` statements for python2 immediately after `1.16` is released will cause annoyances in the first few months bugs are being ironed out.
My 2cents.
On Sun, Nov 4, 2018 at 10:04 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, Nov 4, 2018 at 6:16 PM Stephan Hoyer <shoyer@gmail.com> wrote:
On Sun, Nov 4, 2018 at 10:32 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Hi Chuck,
For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in performance, this may be the best option. All the best,
Marten
I am also leaning towards this right now, depending on how long we plan to wait for releasing 1.16. It will take us at least a little while to sort out performance issues for __array_function__, I'd guess at least a few weeks. Then a blocker still might turn up during the release candidate process (though I think we've found most of the major bugs / downstream issues already through tests on NumPy's dev branch).
My tentative schedule is to branch in about two weeks, then allow 2 weeks of testing for rc1, possibly another two weeks for rc2, and then a final. so possibly about six weeks to final release. That leaves 2 to 4 weeks of slack before 2019.
Overall, it does feels a little misguided to rush in a change as pervasive as __array_function__ for a long term support release. If we exclude __array_function__ I expect the whole release process for 1.16 would go much smoother. We might even try to get 1.17 out faster than usual, so we can minimize the number of additional changes besides __array_function__ and going Python 3 only -- that's already a good bit of change.
I would like to get 1.17 out a bit early. I'm not sure how many backwards incompatible changes we want to have in the first post python2 release. My initial thoughts are to drop Python 2.7 testing, go to C99, and get the new fft in. Beyond that, I'm hesitant to start tearing out all the Python2 special casing in the first new release, although that could certainly be the main task for 1.17 and would clean up the code considerably. It might also be a good time to catch up on changing deprecations to errors. Thoughts on how to proceed are welcome.
Note that if we make this change (reverting __array_function__), we'll need to revisit where we put a few deprecation warnings -- these will need to be restored into function bodies, not their dispatcher functions.
Also: it would be really nice if we get matmul-as-ufunc in before (or at the same time) as __array_function__, so we have a complete story about it being possible to override everything in NumPy. This is another argument for delaying __array_function__, if matmul-as-ufunc can't make it in time for 1.16.
That's two votes for matmul-as-ufunc. How much would it cost to simply make __array_function__ a nop?
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Sun, 04 Nov 2018 17:16:12 -0800, Stephan Hoyer wrote:
On Sun, Nov 4, 2018 at 10:32 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in performance, this may be the best option. All the best,
I am also leaning towards this right now, depending on how long we plan to wait for releasing 1.16. It will take us at least a little while to sort out performance issues for __array_function__, I'd guess at least a few weeks. Then a blocker still might turn up during the release candidate process (though I think we've found most of the major bugs / downstream issues already through tests on NumPy's dev branch).
Just to make sure I understand correctly: the suggestion is to use an environment variable to temporarily toggle the feature, but the plan in the long run will be to have it enabled all the time, correct? Stéfan
On Tue, Nov 6, 2018 at 6:08 PM Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Sun, Nov 4, 2018 at 10:32 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in
On Sun, 04 Nov 2018 17:16:12 -0800, Stephan Hoyer wrote: performance,
this may be the best option. All the best,
I am also leaning towards this right now, depending on how long we plan to wait for releasing 1.16. It will take us at least a little while to sort out performance issues for __array_function__, I'd guess at least a few weeks. Then a blocker still might turn up during the release candidate process (though I think we've found most of the major bugs / downstream issues already through tests on NumPy's dev branch).
Just to make sure I understand correctly: the suggestion is to use an environment variable to temporarily toggle the feature, but the plan in the long run will be to have it enabled all the time, correct?
Yes, exactly. __array_function__ would be opt-in only for 1.16 but enabled by default for 1.17.
On Tue, Nov 6, 2018 at 8:09 PM Stephan Hoyer <shoyer@gmail.com> wrote:
On Tue, Nov 6, 2018 at 6:08 PM Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Sun, Nov 4, 2018 at 10:32 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
For `__array_function__`, there was some discussion in https://github.com/numpy/numpy/issues/12225 that for 1.16 we might want to follow after all Nathaniel's suggestion of using an environment variable or so to opt in (since introspection breaks on python2 with our wrapped implementations). Given also the possibly significant hit in
On Sun, 04 Nov 2018 17:16:12 -0800, Stephan Hoyer wrote: performance,
this may be the best option. All the best,
I am also leaning towards this right now, depending on how long we plan to wait for releasing 1.16. It will take us at least a little while to sort out performance issues for __array_function__, I'd guess at least a few weeks. Then a blocker still might turn up during the release candidate process (though I think we've found most of the major bugs / downstream issues already through tests on NumPy's dev branch).
Just to make sure I understand correctly: the suggestion is to use an environment variable to temporarily toggle the feature, but the plan in the long run will be to have it enabled all the time, correct?
Yes, exactly. __array_function__ would be opt-in only for 1.16 but enabled by default for 1.17.
I've gone ahead and made a pull request making __array_function__ opt-in only for now: https://github.com/numpy/numpy/pull/12362
On 4/11/18 8:04 pm, Charles R Harris wrote:
Hi All,
Time to begin looking forward to the NumPy 1.16.x branch. I think there are three main topics to address:
1. current PRs that need review and merging, 2. critical fixes that need to be made, 3. status of `__array_function__`.
The last probably needs some discussion. `__array_fuction__` seems to be working at this point, but does cause noticeable slowdowns in some function calls. I don't know if those slowdowns are significant in practice, the only way to discover that may be to make the release, or at least thorough testing of the release candidates, but we should at least discuss them and possible workarounds if needed. Trying to have things in good shape is important because 1.16.x will be the last release that supports Python 2.7, and even though we will be maintaining it for the next year, that will be easier if it is stable. As to the first two topics, I think we should try to be conservative at this point and look mostly for bug fixes and documentation updates.
Thoughts?
Chuck
Beyond things with the 1.16 milestone, it would be nice to address the structured array cleanup https://gist.github.com/ahaldane/6cd44886efb449f9c8d5ea012747323b and to get the matmul-as-ufunc https://github.com/numpy/numpy/pull/12219 merged Matti
participants (6)
-
Charles R Harris -
Mark Harfouche -
Marten van Kerkwijk -
Matti Picus -
Stefan van der Walt -
Stephan Hoyer