Hi All,
I will be running the NumPy sprint at Scipy 2017 and I'm trying to put together a suitable list of things to sprint on. In my experience, sprinting on NumPy is hard, enhancements generally need lengthy review and even finding and doing simple bug fixes can take time. What I have in mind at this point, apart from what might be a getting started tutorial, could mostly be classified as janitorial work.
1. Triage issues and close those that no longer apply. This is mind numbing work, but it has been almost three years since the last pass. 2. Move the contents of `numpy/doc` into `doc/source` and make them normal *.rst files. 3. Convert the doctest in `numpy/lib/tests/test_polynomial.py` to regular tests. Might be tricky as it mostly checks print formatting.
On Thu, Jun 29, 2017 at 12:07 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hi All,
I will be running the NumPy sprint at Scipy 2017 and I'm trying to put together a suitable list of things to sprint on. In my experience, sprinting on NumPy is hard, enhancements generally need lengthy review and even finding and doing simple bug fixes can take time. What I have in mind at this point, apart from what might be a getting started tutorial, could mostly be classified as janitorial work.
- Triage issues and close those that no longer apply. This is mind
numbing work, but it has been almost three years since the last pass. 2. Move the contents of `numpy/doc` into `doc/source` and make them normal *.rst files. 3. Convert the doctest in `numpy/lib/tests/test_polynomial.py` to regular tests. Might be tricky as it mostly checks print formatting.
Oops, sent prematurely by accident.
Anyway, I'm looking for suggestions and comments as to things to do that would be useful.
Chuck
On Thu, Jun 29, 2017, at 11:09, Charles R Harris wrote:
On Thu, Jun 29, 2017 at 12:07 PM, Charles R Harris charlesr.harris@gmail.com wrote:>> I will be running the NumPy sprint at Scipy 2017 and I'm trying to
put together a suitable list of things to sprint on. In my experience, sprinting on NumPy is hard, enhancements generally need lengthy review and even finding and doing simple bug fixes can take time. What I have in mind at this point, apart from what might be a getting started tutorial, could mostly be classified as janitorial work.
Here's a random idea: how about building a NumPy gallery? scikit- {image,learn} has it, and while those projects may have more visual datasets, I can imagine something along the lines of Nicolas Rougier's beautiful book: http://www.labri.fr/perso/nrougier/from-python-to-numpy/
Stéfan
On Thu, Jun 29, 2017 at 12:15 PM, Stefan van der Walt stefanv@berkeley.edu wrote:
On Thu, Jun 29, 2017, at 11:09, Charles R Harris wrote:
On Thu, Jun 29, 2017 at 12:07 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
I will be running the NumPy sprint at Scipy 2017 and I'm trying to put together a suitable list of things to sprint on. In my experience, sprinting on NumPy is hard, enhancements generally need lengthy review and even finding and doing simple bug fixes can take time. What I have in mind at this point, apart from what might be a getting started tutorial, could mostly be classified as janitorial work.
Here's a random idea: how about building a NumPy gallery? scikit-{image,learn} has it, and while those projects may have more visual datasets, I can imagine something along the lines of Nicolas Rougier's beautiful book:
So that would be added in the numpy https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org repo?
Chuck
Charles R Harris kirjoitti 29.06.2017 klo 20:45:
Here's a random idea: how about building a NumPy gallery? scikit-{image,learn} has it, and while those projects may have more visual datasets, I can imagine something along the lines of Nicolas Rougier's beautiful book: http://www.labri.fr/perso/nrougier/from-python-to-numpy/ <http://www.labri.fr/perso/nrougier/from-python-to-numpy/>
So that would be added in the numpy https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org repo?
Or https://scipy-cookbook.readthedocs.io/ ? (maybe minus bitrot and images added :)
On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen pav@iki.fi wrote:
Charles R Harris kirjoitti 29.06.2017 klo 20:45:
Here's a random idea: how about building a NumPy gallery? scikit-{image,learn} has it, and while those projects may have more visual datasets, I can imagine something along the lines of Nicolas Rougier's beautiful book: http://www.labri.fr/perso/nrougier/from-python-to-numpy/ <http://www.labri.fr/perso/nrougier/from-python-to-numpy/>
So that would be added in the numpy https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org repo?
Or https://scipy-cookbook.readthedocs.io/ ? (maybe minus bitrot and images added :) _____________________________________
I'd like the numpy.org one. numpy.org is now incredibly sparse and ugly, a gallery would make it look a lot better.
Another idea, from the "deprecate np.matrix" discussion: add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Ralf
Just a heads-up. There is now a sphinx-gallery plugin. Matplotlib and a few other projects have migrated their docs over to use it.
https://sphinx-gallery.readthedocs.io/en/latest/
Cheers! Ben Root
On Sat, Jul 1, 2017 at 7:12 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen pav@iki.fi wrote:
Charles R Harris kirjoitti 29.06.2017 klo 20:45:
Here's a random idea: how about building a NumPy gallery? scikit-{image,learn} has it, and while those projects may have more visual datasets, I can imagine something along the lines of Nicolas Rougier's beautiful book: http://www.labri.fr/perso/nrougier/from-python-to-numpy/ <http://www.labri.fr/perso/nrougier/from-python-to-numpy/>
So that would be added in the numpy https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org repo?
Or https://scipy-cookbook.readthedocs.io/ ? (maybe minus bitrot and images added :) _____________________________________
I'd like the numpy.org one. numpy.org is now incredibly sparse and ugly, a gallery would make it look a lot better.
Another idea, from the "deprecate np.matrix" discussion: add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Ralf
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Updated list below.
On Sat, Jul 1, 2017 at 7:08 PM, Benjamin Root ben.v.root@gmail.com wrote:
Just a heads-up. There is now a sphinx-gallery plugin. Matplotlib and a few other projects have migrated their docs over to use it.
https://sphinx-gallery.readthedocs.io/en/latest/
Cheers! Ben Root
On Sat, Jul 1, 2017 at 7:12 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen pav@iki.fi wrote:
Charles R Harris kirjoitti 29.06.2017 klo 20:45:
Here's a random idea: how about building a NumPy gallery? scikit-{image,learn} has it, and while those projects may have more visual datasets, I can imagine something along the lines of Nicolas Rougier's beautiful book: http://www.labri.fr/perso/nrougier/from-python-to-numpy/ <http://www.labri.fr/perso/nrougier/from-python-to-numpy/>
So that would be added in the numpy https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org repo?
Or https://scipy-cookbook.readthedocs.io/ ? (maybe minus bitrot and images added :) _____________________________________
I'd like the numpy.org one. numpy.org is now incredibly sparse and ugly, a gallery would make it look a lot better.
Another idea, from the "deprecate np.matrix" discussion: add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Putting things together with a few new ideas,
1. add gallery to numpy.org, 2. add extended documentation of '@' operator, 3. make Numpy tests Pytest compatible, 4. add matrix multiplication ufunc.
Any more ideas?
Chuck
On 07/02/2017 10:03 AM, Charles R Harris wrote:
Updated list below.
On Sat, Jul 1, 2017 at 7:08 PM, Benjamin Root <ben.v.root@gmail.com mailto:ben.v.root@gmail.com> wrote:
Just a heads-up. There is now a sphinx-gallery plugin. Matplotlib and a few other projects have migrated their docs over to use it. https://sphinx-gallery.readthedocs.io/en/latest/ <https://sphinx-gallery.readthedocs.io/en/latest/> Cheers! Ben Root On Sat, Jul 1, 2017 at 7:12 AM, Ralf Gommers <ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>> wrote: On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen <pav@iki.fi <mailto:pav@iki.fi>> wrote: Charles R Harris kirjoitti 29.06.2017 klo 20:45: > Here's a random idea: how about building a NumPy gallery? > scikit-{image,learn} has it, and while those projects may have more > visual datasets, I can imagine something along the lines of Nicolas > Rougier's beautiful book: > > http://www.labri.fr/perso/nrougier/from-python-to-numpy/ <http://www.labri.fr/perso/nrougier/from-python-to-numpy/> > <http://www.labri.fr/perso/nrougier/from-python-to-numpy/ <http://www.labri.fr/perso/nrougier/from-python-to-numpy/>> > > > So that would be added in the numpy > <https://github.com/numpy>/numpy.org <http://numpy.org> > <https://github.com/numpy/numpy.org <https://github.com/numpy/numpy.org>> repo? Or https://scipy-cookbook.readthedocs.io/ <https://scipy-cookbook.readthedocs.io/> ? (maybe minus bitrot and images added :) _____________________________________ I'd like the numpy.org <http://numpy.org> one. numpy.org <http://numpy.org> is now incredibly sparse and ugly, a gallery would make it look a lot better. Another idea, from the "deprecate np.matrix" discussion: add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Putting things together with a few new ideas,
- add gallery to numpy.org http://numpy.org,
- add extended documentation of '@' operator,
- make Numpy tests Pytest compatible,
- add matrix multiplication ufunc.
Any more ideas?
The new doctest runner suggested in the printing thread? This is to ignore whitespace and precision in ndarray output.
I can see an argument for distributing it in numpy if it is designed to be specially aware of ndarrays or numpy scalars (eg to test equality between 'wants' and 'got')
Allan
Chuck
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Sun, 2017-07-02 at 10:49 -0400, Allan Haldane wrote:
On 07/02/2017 10:03 AM, Charles R Harris wrote:
Updated list below.
On Sat, Jul 1, 2017 at 7:08 PM, Benjamin Root <ben.v.root@gmail.com mailto:ben.v.root@gmail.com> wrote:
Just a heads-up. There is now a sphinx-gallery plugin. Matplotlib and a few other projects have migrated their docs over to use it.
https://sphinx-gallery.readthedocs.io/en/latest/ https://sphinx-gallery.readthedocs.io/en/latest/
Cheers! Ben Root
On Sat, Jul 1, 2017 at 7:12 AM, Ralf Gommers <ralf.gommers@gmai l.com mailto:ralf.gommers@gmail.com> wrote:
On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen <pav@iki.fi mailto:pav@iki.fi> wrote:
Charles R Harris kirjoitti 29.06.2017 klo 20:45: > Here's a random idea: how about building a NumPy gallery? > scikit-{image,learn} has it, and while those projects may have more > visual datasets, I can imagine something along the lines of Nicolas > Rougier's beautiful book: > > http://www.labri.fr/perso/nrougier/from-python-to -numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/ > <http://www.labri.fr/perso/nrougier/from-python-t o-numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/> > > > So that would be added in the numpy > https://github.com/numpy/numpy.org http://numpy.org > <https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org> repo?
Or https://scipy-cookbook.readthedocs.io/ https://scipy-cookbook.readthedocs.io/ ? (maybe minus bitrot and images added :) _____________________________________
I'd like the numpy.org http://numpy.org one. numpy.org http://numpy.org is now incredibly sparse and ugly, a gallery would make it look a lot better.
Another idea, from the "deprecate np.matrix" discussion: add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Putting things together with a few new ideas,
1. add gallery to numpy.org http://numpy.org, 2. add extended documentation of '@' operator, 3. make Numpy tests Pytest compatible, 4. add matrix multiplication ufunc.
Any more ideas?
The new doctest runner suggested in the printing thread? This is to ignore whitespace and precision in ndarray output.
I can see an argument for distributing it in numpy if it is designed to be specially aware of ndarrays or numpy scalars (eg to test equality between 'wants' and 'got')
I don't really feel it is very numpy specific or should be under the numpy umbrella (I mean if there is no other spot, I guess it could live on the numpy github page). Its about as numpy specific, as the gallery sphinx extension is probably matplotlib specific....
That doesn't mean that it might not be a good sprint, though :).
The question to me is a bit what those who actually go there want from it or do a few people who know numpy/scipy already plan to come? Two years ago, we did not have much of a plan, so it was mostly giving three people or so a bit of a tutorial of how numpy worked internally leading to some bug fixes.
One quick idea that might be nice and dives a bit into the C-layer (might be nice if there is no big topic with a few people working on):
* Find places that should have the new memory overlap detection and implement it there.
If someone who does subclasses/array-likes or so (e.g. like Stefan Hoyer ;)) and is interested, and also we do some teleconferencing/chatting (and I have time).... I might be interested in discussing and possibly trying to develop the new indexer ideas, which I feel are pretty far, but I got stuck on how to get subclasses right.
- Sebastian
Allan
Chuck
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Sun, Jul 2, 2017 at 9:33 AM, Sebastian Berg sebastian@sipsolutions.net wrote:
On Sun, 2017-07-02 at 10:49 -0400, Allan Haldane wrote:
On 07/02/2017 10:03 AM, Charles R Harris wrote:
Updated list below.
On Sat, Jul 1, 2017 at 7:08 PM, Benjamin Root <ben.v.root@gmail.com
mailto:ben.v.root@gmail.com> wrote:
Just a heads-up. There is now a sphinx-gallery plugin.
Matplotlib and a few other projects have migrated their docs over to use it.
https://sphinx-gallery.readthedocs.io/en/latest/ <https://sphinx-gallery.readthedocs.io/en/latest/> Cheers! Ben Root On Sat, Jul 1, 2017 at 7:12 AM, Ralf Gommers <ralf.gommers@gmai
l.com mailto:ralf.gommers@gmail.com> wrote:
On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen <pav@iki.fi <mailto:pav@iki.fi>> wrote: Charles R Harris kirjoitti 29.06.2017 klo 20:45: > Here's a random idea: how about building a NumPy
gallery? > scikit-{image,learn} has it, and while those projects may have more > visual datasets, I can imagine something along the lines of Nicolas > Rougier's beautiful book: > > http://www.labri.fr/perso/nrougier/from-python-to -numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/ > <http://www.labri.fr/perso/nrougier/from-python-t o-numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/> > > > So that would be added in the numpy > https://github.com/numpy/numpy.org http://numpy.org > <https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org> repo?
Or https://scipy-cookbook.readthedocs.io/ <https://scipy-cookbook.readthedocs.io/> ? (maybe minus bitrot and images added :) _____________________________________ I'd like the numpy.org <http://numpy.org> one. numpy.org <http://numpy.org> is now incredibly sparse and ugly, a
gallery would make it look a lot better.
Another idea, from the "deprecate np.matrix" discussion:
add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Putting things together with a few new ideas,
- add gallery to numpy.org http://numpy.org,
- add extended documentation of '@' operator,
- make Numpy tests Pytest compatible,
- add matrix multiplication ufunc.
Any more ideas?
The new doctest runner suggested in the printing thread? This is to ignore whitespace and precision in ndarray output.
I can see an argument for distributing it in numpy if it is designed to be specially aware of ndarrays or numpy scalars (eg to test equality between 'wants' and 'got')
I don't really feel it is very numpy specific or should be under the numpy umbrella (I mean if there is no other spot, I guess it could live on the numpy github page). Its about as numpy specific, as the gallery sphinx extension is probably matplotlib specific....
That doesn't mean that it might not be a good sprint, though :).
The question to me is a bit what those who actually go there want from it or do a few people who know numpy/scipy already plan to come? Two years ago, we did not have much of a plan, so it was mostly giving three people or so a bit of a tutorial of how numpy worked internally leading to some bug fixes.
One quick idea that might be nice and dives a bit into the C-layer (might be nice if there is no big topic with a few people working on):
- Find places that should have the new memory overlap detection and implement it there.
If someone who does subclasses/array-likes or so (e.g. like Stefan Hoyer ;)) and is interested, and also we do some teleconferencing/chatting (and I have time).... I might be interested in discussing and possibly trying to develop the new indexer ideas, which I feel are pretty far, but I got stuck on how to get subclasses right.
- Sebastian
I've opened an issue for Pytests https://github.com/numpy/numpy/issues/9352 and given it a "Scipy2017 Sprint" label. I'd be much obliged if the folks with suggestions here would open other issues and also label them with "Scipy2017 Sprint". Note that these issues are not Scipy 2017 specific, they could be used in other contexts, but I thought is might be useful to collect them in one spot and give them some structure together with suggestions on how to proceed.
Ralf, you have made several previous suggestion on bringing over some to the scipy tests to numpy, to include documentation testing. Were there any other tests we should look into?
Chuck
On Mon, Jul 3, 2017 at 7:01 AM, Charles R Harris charlesr.harris@gmail.com wrote:
On Sun, Jul 2, 2017 at 9:33 AM, Sebastian Berg <sebastian@sipsolutions.net
wrote:
On Sun, 2017-07-02 at 10:49 -0400, Allan Haldane wrote:
On 07/02/2017 10:03 AM, Charles R Harris wrote:
Updated list below.
On Sat, Jul 1, 2017 at 7:08 PM, Benjamin Root <ben.v.root@gmail.com
mailto:ben.v.root@gmail.com> wrote:
Just a heads-up. There is now a sphinx-gallery plugin.
Matplotlib and a few other projects have migrated their docs over to use it.
https://sphinx-gallery.readthedocs.io/en/latest/ <https://sphinx-gallery.readthedocs.io/en/latest/> Cheers! Ben Root On Sat, Jul 1, 2017 at 7:12 AM, Ralf Gommers <ralf.gommers@gmai
l.com mailto:ralf.gommers@gmail.com> wrote:
On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen <pav@iki.fi <mailto:pav@iki.fi>> wrote: Charles R Harris kirjoitti 29.06.2017 klo 20:45: > Here's a random idea: how about building a NumPy
gallery? > scikit-{image,learn} has it, and while those projects may have more > visual datasets, I can imagine something along the lines of Nicolas > Rougier's beautiful book: > > http://www.labri.fr/perso/nrougier/from-python-to -numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/ > <http://www.labri.fr/perso/nrougier/from-python-t o-numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/> > > > So that would be added in the numpy > https://github.com/numpy/numpy.org http://numpy.org > <https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org> repo?
Or https://scipy-cookbook.readthedocs.io/ <https://scipy-cookbook.readthedocs.io/> ? (maybe minus bitrot and images added :) _____________________________________ I'd like the numpy.org <http://numpy.org> one. numpy.org <http://numpy.org> is now incredibly sparse and ugly, a
gallery would make it look a lot better.
Another idea, from the "deprecate np.matrix" discussion:
add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Putting things together with a few new ideas,
- add gallery to numpy.org http://numpy.org,
- add extended documentation of '@' operator,
- make Numpy tests Pytest compatible,
- add matrix multiplication ufunc.
Any more ideas?
The new doctest runner suggested in the printing thread? This is to ignore whitespace and precision in ndarray output.
I can see an argument for distributing it in numpy if it is designed to be specially aware of ndarrays or numpy scalars (eg to test equality between 'wants' and 'got')
I don't really feel it is very numpy specific or should be under the numpy umbrella (I mean if there is no other spot, I guess it could live on the numpy github page). Its about as numpy specific, as the gallery sphinx extension is probably matplotlib specific....
That doesn't mean that it might not be a good sprint, though :).
The question to me is a bit what those who actually go there want from it or do a few people who know numpy/scipy already plan to come? Two years ago, we did not have much of a plan, so it was mostly giving three people or so a bit of a tutorial of how numpy worked internally leading to some bug fixes.
One quick idea that might be nice and dives a bit into the C-layer (might be nice if there is no big topic with a few people working on):
- Find places that should have the new memory overlap detection and implement it there.
If someone who does subclasses/array-likes or so (e.g. like Stefan Hoyer ;)) and is interested, and also we do some teleconferencing/chatting (and I have time).... I might be interested in discussing and possibly trying to develop the new indexer ideas, which I feel are pretty far, but I got stuck on how to get subclasses right.
- Sebastian
I've opened an issue for Pytests https://github.com/numpy/numpy/issues/9352 and given it a "Scipy2017 Sprint" label. I'd be much obliged if the folks with suggestions here would open other issues and also label them with "Scipy2017 Sprint". Note that these issues are not Scipy 2017 specific, they could be used in other contexts, but I thought is might be useful to collect them in one spot and give them some structure together with suggestions on how to proceed.
Ralf, you have made several previous suggestion on bringing over some to the scipy tests to numpy, to include documentation testing. Were there any other tests we should look into?
Better platform test coverage would be a useful topic if someone is willing to work on that. NumPy needs OS X testing enabled on TravisCI, SciPy needs OS X and a 32-bit test (steal from NumPy). And if someone really feels ambitious: replace ATLAS by OpenBLAS in one of the test matrix entries.
Ralf
Note that TravisCI does not yet have official Python support on Mac OS X,
https://github.com/travis-ci/travis-ci/issues/2312
I believe it is possible to do anyway by faking it under another setting (e.g. pretend to be a generic language build, and use the system Python or install your own specific version of Python as needed), so that may be worth trying during a sprint.
Peter
On Wed, Jul 5, 2017 at 10:43 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
Better platform test coverage would be a useful topic if someone is willing to work on that. NumPy needs OS X testing enabled on TravisCI, SciPy needs OS X and a 32-bit test (steal from NumPy). And if someone really feels ambitious: replace ATLAS by OpenBLAS in one of the test matrix entries.
Ralf
On Wed, Jul 5, 2017 at 10:14 PM, Peter Cock p.j.a.cock@googlemail.com wrote:
Note that TravisCI does not yet have official Python support on Mac OS X,
https://github.com/travis-ci/travis-ci/issues/2312
I believe it is possible to do anyway by faking it under another setting (e.g. pretend to be a generic language build, and use the system Python or install your own specific version of Python as needed), so that may be worth trying during a sprint.
That approach has worked reliably for https://github.com/MacPython/numpy-wheels for a while now, so should be straightforward.
Ralf
Peter
On Wed, Jul 5, 2017 at 10:43 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
Better platform test coverage would be a useful topic if someone is
willing
to work on that. NumPy needs OS X testing enabled on TravisCI, SciPy
needs
OS X and a 32-bit test (steal from NumPy). And if someone really feels ambitious: replace ATLAS by OpenBLAS in one of the test matrix entries.
Ralf
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Wed, Jul 5, 2017 at 11:25 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jul 5, 2017 at 10:14 PM, Peter Cock p.j.a.cock@googlemail.com wrote:
Note that TravisCI does not yet have official Python support on Mac OS X,
https://github.com/travis-ci/travis-ci/issues/2312
I believe it is possible to do anyway by faking it under another setting (e.g. pretend to be a generic language build, and use the system Python or install your own specific version of Python as needed), so that may be worth trying during a sprint.
That approach has worked reliably for https://github.com/MacPython/numpy-wheels for a while now, so should be straightforward.
And https://travis-ci.org/MacPython/scipy-wheels where we are testing OSX, 64 and 32 bit manylinux builds daily. That didn't catch the recent ndimage error because I'd disabled the 32-bit builds there.
Numpy, scipy, and a fairly large number of other projects use https://github.com/matthew-brett/multibuild to set up builds in this way for manylinux, OSX and (with a bit more effort) Windows.
Cheers,
Matthew
On Wed, Jul 5, 2017 at 11:25 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jul 5, 2017 at 10:14 PM, Peter Cock p.j.a.cock@googlemail.com wrote:
Note that TravisCI does not yet have official Python support on Mac OS X,
https://github.com/travis-ci/travis-ci/issues/2312
I believe it is possible to do anyway by faking it under another setting (e.g. pretend to be a generic language build, and use the system Python or install your own specific version of Python as needed), so that may be worth trying during a sprint.
That approach has worked reliably for https://github.com/MacPython/numpy-wheels for a while now, so should be straightforward.
Ralf
Thanks for that link - I'm going off topic but the MacPython wiki page goes into more background about how they build wheels for PyPI which I'm very interested to read up on:
https://github.com/MacPython/wiki/wiki/Spinning-wheels
Peter
On Wed, Jul 5, 2017 at 11:31 AM, Peter Cock p.j.a.cock@googlemail.com wrote:
On Wed, Jul 5, 2017 at 11:25 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jul 5, 2017 at 10:14 PM, Peter Cock p.j.a.cock@googlemail.com wrote:
Note that TravisCI does not yet have official Python support on Mac OS X,
https://github.com/travis-ci/travis-ci/issues/2312
I believe it is possible to do anyway by faking it under another setting (e.g. pretend to be a generic language build, and use the system Python or install your own specific version of Python as needed), so that may be worth trying during a sprint.
That approach has worked reliably for https://github.com/MacPython/numpy-wheels for a while now, so should be straightforward.
Ralf
Thanks for that link - I'm going off topic but the MacPython wiki page goes into more background about how they build wheels for PyPI which I'm very interested to read up on:
Yes, you'll see that the multibuild framework that numpy and scipy uses, includes utilities to download Python.org Python and build against that, in Spinning-wheels fashion,
Cheers,
Matthew
On Wed, Jul 5, 2017 at 10:43 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Mon, Jul 3, 2017 at 7:01 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, Jul 2, 2017 at 9:33 AM, Sebastian Berg < sebastian@sipsolutions.net> wrote:
On Sun, 2017-07-02 at 10:49 -0400, Allan Haldane wrote:
On 07/02/2017 10:03 AM, Charles R Harris wrote:
Updated list below.
On Sat, Jul 1, 2017 at 7:08 PM, Benjamin Root <ben.v.root@gmail.com
mailto:ben.v.root@gmail.com> wrote:
Just a heads-up. There is now a sphinx-gallery plugin.
Matplotlib and a few other projects have migrated their docs over to use it.
https://sphinx-gallery.readthedocs.io/en/latest/ <https://sphinx-gallery.readthedocs.io/en/latest/> Cheers! Ben Root On Sat, Jul 1, 2017 at 7:12 AM, Ralf Gommers <ralf.gommers@gmai
l.com mailto:ralf.gommers@gmail.com> wrote:
On Fri, Jun 30, 2017 at 6:50 AM, Pauli Virtanen <pav@iki.fi <mailto:pav@iki.fi>> wrote: Charles R Harris kirjoitti 29.06.2017 klo 20:45: > Here's a random idea: how about building a NumPy
gallery? > scikit-{image,learn} has it, and while those projects may have more > visual datasets, I can imagine something along the lines of Nicolas > Rougier's beautiful book: > > http://www.labri.fr/perso/nrougier/from-python-to -numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/ > <http://www.labri.fr/perso/nrougier/from-python-t o-numpy/ http://www.labri.fr/perso/nrougier/from-python-to-nump y/> > > > So that would be added in the numpy > https://github.com/numpy/numpy.org http://numpy.org > <https://github.com/numpy/numpy.org https://github.com/numpy/numpy.org> repo?
Or https://scipy-cookbook.readthedocs.io/ <https://scipy-cookbook.readthedocs.io/> ? (maybe minus bitrot and images added :) _____________________________________ I'd like the numpy.org <http://numpy.org> one. numpy.org <http://numpy.org> is now incredibly sparse and ugly, a
gallery would make it look a lot better.
Another idea, from the "deprecate np.matrix" discussion:
add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section.
Putting things together with a few new ideas,
- add gallery to numpy.org http://numpy.org,
- add extended documentation of '@' operator,
- make Numpy tests Pytest compatible,
- add matrix multiplication ufunc.
Any more ideas?
The new doctest runner suggested in the printing thread? This is to ignore whitespace and precision in ndarray output.
I can see an argument for distributing it in numpy if it is designed to be specially aware of ndarrays or numpy scalars (eg to test equality between 'wants' and 'got')
I don't really feel it is very numpy specific or should be under the numpy umbrella (I mean if there is no other spot, I guess it could live on the numpy github page). Its about as numpy specific, as the gallery sphinx extension is probably matplotlib specific....
That doesn't mean that it might not be a good sprint, though :).
The question to me is a bit what those who actually go there want from it or do a few people who know numpy/scipy already plan to come? Two years ago, we did not have much of a plan, so it was mostly giving three people or so a bit of a tutorial of how numpy worked internally leading to some bug fixes.
One quick idea that might be nice and dives a bit into the C-layer (might be nice if there is no big topic with a few people working on):
- Find places that should have the new memory overlap detection and implement it there.
If someone who does subclasses/array-likes or so (e.g. like Stefan Hoyer ;)) and is interested, and also we do some teleconferencing/chatting (and I have time).... I might be interested in discussing and possibly trying to develop the new indexer ideas, which I feel are pretty far, but I got stuck on how to get subclasses right.
- Sebastian
I've opened an issue for Pytests https://github.com/numpy/numpy/issues/9352 and given it a "Scipy2017 Sprint" label. I'd be much obliged if the folks with suggestions here would open other issues and also label them with "Scipy2017 Sprint". Note that these issues are not Scipy 2017 specific, they could be used in other contexts, but I thought is might be useful to collect them in one spot and give them some structure together with suggestions on how to proceed.
Ralf, you have made several previous suggestion on bringing over some to the scipy tests to numpy, to include documentation testing. Were there any other tests we should look into?
Better platform test coverage would be a useful topic if someone is willing to work on that. NumPy needs OS X testing enabled on TravisCI, SciPy needs OS X and a 32-bit test (steal from NumPy). And if someone really feels ambitious: replace ATLAS by OpenBLAS in one of the test matrix entries.
I can help with that, especially OpenBLAS. Though I would not mind working on something else than packaging :)
David
Ralf
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Lots of good ideas here. It would help if issues were opened for them and flagged with the sprint label. I'll be doing some myself, but I'm not as intimately familiar with some of the topics as the proposers are.
<snip>
Chuck
On Sun, Jul 2, 2017 at 8:33 AM Sebastian Berg sebastian@sipsolutions.net wrote:
If someone who does subclasses/array-likes or so (e.g. like Stefan Hoyer ;)) and is interested, and also we do some teleconferencing/chatting (and I have time).... I might be interested in discussing and possibly trying to develop the new indexer ideas, which I feel are pretty far, but I got stuck on how to get subclasses right.
I am off course very happy to discuss this (online or via teleconference, sadly I won't be at scipy), but to be clear I use array likes, not subclasses. I think Marten van Kerkwijk is the last one who thinks that is still a good idea :).
On Mon, Jul 3, 2017 at 4:27 PM, Stephan Hoyer shoyer@gmail.com wrote:
If someone who does subclasses/array-likes or so (e.g. like Stefan
Hoyer ;)) and is interested, and also we do some teleconferencing/chatting (and I have time).... I might be interested in discussing and possibly trying to develop the new indexer ideas, which I feel are pretty far, but I got stuck on how to get subclasses right.
I am off course very happy to discuss this (online or via teleconference, sadly I won't be at scipy), but to be clear I use array likes, not subclasses. I think Marten van Kerkwijk is the last one who thinks that is still a good idea :).
Indeed -- I thought the community more or less had decided that duck-typing was THE way to make something that could be plugged in where a numpy array is expected.
Along those lines, there was some discussion of having a set of utilities (or maybe eve3n an ABC?) that would make it easier to create a ndarray-like object.
That is, the boilerplate needed for multi-dimensional indexing and slicing, etc...
That could be a nice little sprint-able project.
-CHB
On Wed, Jul 5, 2017 at 10:40 AM, Chris Barker chris.barker@noaa.gov wrote:
Along those lines, there was some discussion of having a set of utilities (or maybe eve3n an ABC?) that would make it easier to create a ndarray-like object.
That is, the boilerplate needed for multi-dimensional indexing and slicing, etc...
That could be a nice little sprint-able project.
Indeed. Let me highlight a few mixins https://github.com/pydata/xarray/blob/6a20f917041abf53bcb35e210d59f5b331211012/xarray/core/utils.py#L381-L425 that I wrote for xarray that might be more broadly useful. The challenge here is that there are quite a few different meanings to "ndarray-like", so mixins really need to be mix-and-match-able. But at least defining a base list of methods to implement/override would be useful.
In NumPy, this could go along with NDArrayOperatorsMixins in numpy/lib/mixins.py https://github.com/numpy/numpy/blob/14cd918c651d72f4c2a8681093e114f01d5bdc36/numpy/lib/mixins.py
On 5 Jul 2017, at 19:05, Stephan Hoyer shoyer@gmail.com wrote:
On Wed, Jul 5, 2017 at 10:40 AM, Chris Barker <chris.barker@noaa.gov mailto:chris.barker@noaa.gov> wrote: Along those lines, there was some discussion of having a set of utilities (or maybe eve3n an ABC?) that would make it easier to create a ndarray-like object.
That is, the boilerplate needed for multi-dimensional indexing and slicing, etc...
That could be a nice little sprint-able project.
Indeed. Let me highlight a few mixins https://github.com/pydata/xarray/blob/6a20f917041abf53bcb35e210d59f5b331211012/xarray/core/utils.py#L381-L425 that I wrote for xarray that might be more broadly useful. The challenge here is that there are quite a few different meanings to "ndarray-like", so mixins really need to be mix-and-match-able. But at least defining a base list of methods to implement/override would be useful.
In NumPy, this could go along with NDArrayOperatorsMixins in numpy/lib/mixins.py https://github.com/numpy/numpy/blob/14cd918c651d72f4c2a8681093e114f01d5bdc36/numpy/lib/mixins.py
Slightly off topic, but as someone who has just spent a fair amount of time implementing various subclasses of nd-array, I am interested (and a little concerned), that the consensus is not to use them. Is there anything available which explains why this is the case and what the alternatives are?
Ben
On Thu, Jul 6, 2017 at 4:42 AM, Ben Rowland bennyrowland@mac.com wrote:
Slightly off topic, but as someone who has just spent a fair amount of time implementing various subclasses of nd-array, I am interested (and a little concerned), that the consensus is not to use them. Is there anything available which explains why this is the case and what the alternatives are?
Writing such docs (especially to explain how to write array-like objects that aren't subclasses) would be another good topic for the sprint ;).
But more seriously: numpy.ndarray subclasses are supported, but inherently error prone, because we don't have a well defined subclassing API. As Martin will attest, this means seemingly harmless internal refactoring in NumPy has a tendency to break downstream subclasses, which often unintentionally end up relying on untested implementation details.
This is particularly problematic when subclasses are implemented in a different code-base, as is the case for user subclasses of numpy.ndarray. Due to diligent testing efforts, we often (but not always) catch these issues before making a release, but the process is inherently error prone. Writing NumPy functionality in a manner that is robust to all possible subclassing approaches turns out to be very difficult (nearly impossible).
This is actually a classic OOP problem, e.g., see https://en.wikipedia.org/wiki/Composition_over_inheritance
On Wed, Jul 5, 2017 at 11:05 AM, Stephan Hoyer shoyer@gmail.com wrote:
That is, the boilerplate needed for multi-dimensional indexing and
slicing, etc...
That could be a nice little sprint-able project.
Indeed. Let me highlight a few mixins https://github.com/pydata/xarray/blob/6a20f917041abf53bcb35e210d59f5b331211012/xarray/core/utils.py#L381-L425 that I wrote for xarray that might be more broadly useful.
At a quick glance, that is exactly the kind of ting I had in mind.
The challenge here is that there are quite a few different meanings to
"ndarray-like", so mixins really need to be mix-and-match-able.
exactly!
But at least defining a base list of methods to implement/override would be useful.
With sample implementations, even... at last of parts of it -- I'm thinking things like parsing out the indexes/slices in __getitem__ -- that sort of thing.
In NumPy, this could go along with NDArrayOperatorsMixins in numpy/lib/mixins.py https://github.com/numpy/numpy/blob/14cd918c651d72f4c2a8681093e114f01d5bdc36/numpy/lib/mixins.py
Yes! I had no idea that existed.
-CHB
On Thu, Jul 6, 2017 at 9:42 AM, Chris Barker chris.barker@noaa.gov wrote:
In NumPy, this could go along with NDArrayOperatorsMixins in
numpy/lib/mixins.py https://github.com/numpy/numpy/blob/14cd918c651d72f4c2a8681093e114f01d5bdc36/numpy/lib/mixins.py
Yes! I had no idea that existed.
It's brand new for NumPy 1.13 :). I wrote it to go along with __array_ufunc__.
Hi All,
I doubt I'm really the last one thinking ndarray subclassing is a good idea, but as that was stated, I feel I should at least pipe in. It seems to me there is both a perceived problem -- with the two subclasses that numpy provides -- `matrix` and `MaskedArray` -- both being problematic in ways that seem to me to have very little to do with subclassing being a bad idea, and a real one following from the fact that numpy was written at a time when python's inheritance system was not as well developed as it is now.
Though based on my experience with Quantity, I'd also argue that the more annoying problems are not so much with `ndarray` itself, but rather with the helper functions. Ufuncs were not so bad -- they really just needed a better override mechanism, which __array_ufunc__ now provides -- but for quite a few of the other functions subclassing was clearly an afterthought. Indeed, `MaskedArray` provides a nice example of this, with its many special `np.ma.<function>` routines, providing huge duplication and thus lots of duplicated bugs (which Eric has been patiently fixing...). Indeed, `MaskedArray` is also a much better example than ndarrat of a class that is really hard to subclass (even though, conceptually, it should be a far easier one).
All that said, duck-type arrays make a lot of sense, and e.g. the slicing and shaping methods are easily emulated, especially if one's underlying data are stored in `ndarray`. For astropy's version of a relevant mixin, see http://docs.astropy.org/en/stable/api/astropy.utils.misc.ShapedLikeNDArray.h...
All the best,
Marten
On Fri, Jul 7, 2017 at 4:27 PM, Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Hi All,
I doubt I'm really the last one thinking ndarray subclassing is a good idea, but as that was stated, I feel I should at least pipe in. It seems to me there is both a perceived problem -- with the two subclasses that numpy provides -- `matrix` and `MaskedArray` -- both being problematic in ways that seem to me to have very little to do with subclassing being a bad idea, and a real one following from the fact that numpy was written at a time when python's inheritance system was not as well developed as it is now.
Though based on my experience with Quantity, I'd also argue that the more annoying problems are not so much with `ndarray` itself, but rather with the helper functions. Ufuncs were not so bad -- they really just needed a better override mechanism, which __array_ufunc__ now provides -- but for quite a few of the other functions subclassing was clearly an afterthought. Indeed, `MaskedArray` provides a nice example of this, with its many special `np.ma.<function>` routines, providing huge duplication and thus lots of duplicated bugs (which Eric has been patiently fixing...). Indeed, `MaskedArray` is also a much better example than ndarrat of a class that is really hard to subclass (even though, conceptually, it should be a far easier one).
All that said, duck-type arrays make a lot of sense, and e.g. the slicing and shaping methods are easily emulated, especially if one's underlying data are stored in `ndarray`. For astropy's version of a relevant mixin, see http://docs.astropy.org/en/stable/api/astropy.utils.misc. ShapedLikeNDArray.html
My biggest problem with subclassing as it exists now is that they don't survive the first encounter with np.asarray (or np.array). So much code written to work with numpy uses that as a bandaid (for e.g. handling lists) that in my experience it's 50/50 whether passing a subclass to a function will actually behave as expected--even if there's no good reason it shouldn't.
Ryan
Hi Ryan,
Indeed, the liberal use of `np.asarray` is one of the main reason the helper routines are relatively annoying. Of course, that is not an argument for using duck-types over subclasses: those wouldn't even survive `asanyarray` (which many numpy routines now have moved to).
All the best,
Marten
On Fri, Jul 7, 2017 at 6:42 PM, Ryan May rmay31@gmail.com wrote:
On Fri, Jul 7, 2017 at 4:27 PM, Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Hi All,
I doubt I'm really the last one thinking ndarray subclassing is a good idea, but as that was stated, I feel I should at least pipe in. It seems to me there is both a perceived problem -- with the two subclasses that numpy provides -- `matrix` and `MaskedArray` -- both being problematic in ways that seem to me to have very little to do with subclassing being a bad idea, and a real one following from the fact that numpy was written at a time when python's inheritance system was not as well developed as it is now.
Though based on my experience with Quantity, I'd also argue that the more annoying problems are not so much with `ndarray` itself, but rather with the helper functions. Ufuncs were not so bad -- they really just needed a better override mechanism, which __array_ufunc__ now provides -- but for quite a few of the other functions subclassing was clearly an afterthought. Indeed, `MaskedArray` provides a nice example of this, with its many special `np.ma.<function>` routines, providing huge duplication and thus lots of duplicated bugs (which Eric has been patiently fixing...). Indeed, `MaskedArray` is also a much better example than ndarrat of a class that is really hard to subclass (even though, conceptually, it should be a far easier one).
All that said, duck-type arrays make a lot of sense, and e.g. the slicing and shaping methods are easily emulated, especially if one's underlying data are stored in `ndarray`. For astropy's version of a relevant mixin, see http://docs.astropy.org/en/stable/api/astropy.utils.misc.Sha pedLikeNDArray.html
My biggest problem with subclassing as it exists now is that they don't survive the first encounter with np.asarray (or np.array). So much code written to work with numpy uses that as a bandaid (for e.g. handling lists) that in my experience it's 50/50 whether passing a subclass to a function will actually behave as expected--even if there's no good reason it shouldn't.
as a downstream developer: The problem is that we cannot trust any array subclass or anything that pretends to be like an array. Even asarray is letting already too many things go through. We would need an indication or guarantee for the behavior to quack in the correct way, otherwise it is very difficult to write code that would work for various subclasses.
(even in the simplest case, writing code that works for matrix and arrays beyond a few lines is getting difficult.)
scipy.stats.mstats is largely not code duplication, it needs to handle the mask (although the nan versions in scipy.stats are catching up).
Josef
Ryan
-- Ryan May
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Will it be possible for people not at SciPy to participate ? The timing looks I could finally spend some time on numpy again this year
David
On Thu, Jun 29, 2017 at 7:07 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
Hi All,
I will be running the NumPy sprint at Scipy 2017 and I'm trying to put together a suitable list of things to sprint on. In my experience, sprinting on NumPy is hard, enhancements generally need lengthy review and even finding and doing simple bug fixes can take time. What I have in mind at this point, apart from what might be a getting started tutorial, could mostly be classified as janitorial work.
- Triage issues and close those that no longer apply. This is mind
numbing work, but it has been almost three years since the last pass. 2. Move the contents of `numpy/doc` into `doc/source` and make them normal *.rst files. 3. Convert the doctest in `numpy/lib/tests/test_polynomial.py` to regular tests. Might be tricky as it mostly checks print formatting.
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion