Development workflow (not git tutorial)
Hi, What is a sensible way to work on (modify, compile, and test) numpy? There is documentation about "contributing to numpy" at: http://docs.scipy.org/doc/numpy-dev/dev/index.html and: http://docs.scipy.org/doc/numpy-dev/dev/gitwash/development_workflow.html but these are entirely focused on using git. I have no problem with that aspect. It is building and testing that I am looking for the Right Way to do. My current approach is to build an empty virtualenv, pip install nose, and from the numpy root directory do "python setup.py build_ext --inplace" and "python -c 'import numpy; numpy.test()'". This works, for my stock system python, though I get a lot of weird messages suggesting distutils problems (for example "python setup.py develop", although suggested by setup.py itself, claims that "develop" is not a command). But I don't know how (for example) to test with python3 without starting from a separate clean source tree. What do you recommend: use virtualenvs? Is building inplace the way to go? Is there a better way to run all tests? Are there other packages that should go into the virtualenv? What is the best way to test on multiple python versions? Switch cleanly between feature branches? Surely I can't be the only person wishing for advice on a sensible way to work with an in-development version of numpy? Perhaps this would be a good addition to CONTRIBUTING.md or the website? Thanks, Anne
On Do, 2015-08-13 at 15:52 +0000, Anne Archibald wrote:
Hi,
What is a sensible way to work on (modify, compile, and test) numpy?
There is documentation about "contributing to numpy" at: http://docs.scipy.org/doc/numpy-dev/dev/index.html
and: http://docs.scipy.org/doc/numpy-dev/dev/gitwash/development_workflow.html
but these are entirely focused on using git. I have no problem with that aspect. It is building and testing that I am looking for the Right Way to do.
My current approach is to build an empty virtualenv, pip install nose, and from the numpy root directory do "python setup.py build_ext --inplace" and "python -c 'import numpy; numpy.test()'". This works, for my stock system python, though I get a lot of weird messages suggesting distutils problems (for example "python setup.py develop", although suggested by setup.py itself, claims that "develop" is not a command). But I don't know how (for example) to test with python3 without starting from a separate clean source tree.
We have the `runtests.py` script which will do exactly this (don't think it gives lots of weird warnings normally). I think that is the only real tip I can give. - Sebastian
What do you recommend: use virtualenvs? Is building inplace the way to go? Is there a better way to run all tests? Are there other packages that should go into the virtualenv? What is the best way to test on multiple python versions? Switch cleanly between feature branches?
Surely I can't be the only person wishing for advice on a sensible way to work with an in-development version of numpy? Perhaps this would be a good addition to CONTRIBUTING.md or the website?
Thanks, Anne _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Thu, Aug 13, 2015 at 10:00 AM, Sebastian Berg <sebastian@sipsolutions.net
wrote:
On Do, 2015-08-13 at 15:52 +0000, Anne Archibald wrote:
Hi,
What is a sensible way to work on (modify, compile, and test) numpy?
There is documentation about "contributing to numpy" at: http://docs.scipy.org/doc/numpy-dev/dev/index.html
and:
http://docs.scipy.org/doc/numpy-dev/dev/gitwash/development_workflow.html
but these are entirely focused on using git. I have no problem with that aspect. It is building and testing that I am looking for the Right Way to do.
My current approach is to build an empty virtualenv, pip install nose, and from the numpy root directory do "python setup.py build_ext --inplace" and "python -c 'import numpy; numpy.test()'". This works, for my stock system python, though I get a lot of weird messages suggesting distutils problems (for example "python setup.py develop", although suggested by setup.py itself, claims that "develop" is not a command). But I don't know how (for example) to test with python3 without starting from a separate clean source tree.
We have the `runtests.py` script which will do exactly this (don't think it gives lots of weird warnings normally). I think that is the only real tip I can give.
+1 for `runtests.py`. Do `python runtests.py --help` to get started. If you want another python, say 3.5, `python3.5 runtests.py`. Chuck
On 2015-08-13 08:52:22, Anne Archibald <peridot.faceted@gmail.com> wrote:
My current approach is to build an empty virtualenv, pip install nose, and from the numpy root directory do "python setup.py build_ext --inplace" and "python -c 'import numpy; numpy.test()'". This works, for my stock system python, though I get a lot of weird messages suggesting distutils problems (for example "python setup.py develop", although suggested by setup.py itself, claims that "develop" is not a command). But I don't know how (for example) to test with python3 without starting from a separate clean source tree.
Nowadays, you can use pip install -e . to install an in-place "editable" version of numpy. This should also execute "build_ext" for you. Stéfan
On Thu, Aug 13, 2015 at 11:25 AM, Stefan van der Walt <stefanv@berkeley.edu> wrote:
(for example "python setup.py develop", although suggested by setup.py itself, claims that "develop" is not a command).
develop is a command provided by setuptools, not distutils itself. I find it absolutely invaluable -- it is THE way to go when actively working on any package under development. if numpy doesn't currently use setuptools, it probably should (though maybe it's gets messy with numpy's distutils extensions...) Nowadays, you can use
pip install -e .
pip "injects" setuptools into the mix -- so this may be develope mode with a different name. but yes, a fine option for a package that doesn't use setuptools out of the box. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I used to be a huge advocate for the "develop" mode, but not anymore. I have run into way too many Heisenbugs that would clear up if I nuked my source tree and re-clone. I should also note that there is currently an open issue with "pip install -e" and namespace packages. This has been reported to matplotlib with regards to mpl_toolkits. Essentially, if you have namespace packages, it doesn't get installed correctly in this mode, and python won't find them. On Fri, Aug 14, 2015 at 12:12 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Thu, Aug 13, 2015 at 11:25 AM, Stefan van der Walt < stefanv@berkeley.edu> wrote:
(for example "python setup.py develop", although suggested by setup.py itself, claims that "develop" is not a command).
develop is a command provided by setuptools, not distutils itself.
I find it absolutely invaluable -- it is THE way to go when actively working on any package under development.
if numpy doesn't currently use setuptools, it probably should (though maybe it's gets messy with numpy's distutils extensions...)
Nowadays, you can use
pip install -e .
pip "injects" setuptools into the mix -- so this may be develope mode with a different name. but yes, a fine option for a package that doesn't use setuptools out of the box.
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On 2015-08-14 10:08:11, Benjamin Root <ben.root@ou.edu> wrote:
I should also note that there is currently an open issue with "pip install -e" and namespace packages. This has been reported to matplotlib with regards to mpl_toolkits. Essentially, if you have namespace packages, it doesn't get installed correctly in this mode, and python won't find them.
There are lots of issues with namespace packages, which is why what used to be scikits.learn and scikits.image are now all standalone packages. Perhaps mpl_toolkits should think of becoming mpl_3d, mpl_basemaps, etc.? Stéfan
On 08/13/2015 11:52 AM, Anne Archibald wrote:
Hi,
What is a sensible way to work on (modify, compile, and test) numpy?
There is documentation about "contributing to numpy" at: http://docs.scipy.org/doc/numpy-dev/dev/index.html and: http://docs.scipy.org/doc/numpy-dev/dev/gitwash/development_workflow.html but these are entirely focused on using git. I have no problem with that aspect. It is building and testing that I am looking for the Right Way to do.
Related to this, does anyone know how to debug numpy in gdb with proper symbols/source lines, like I can do with other C extensions? I've tried modifying numpy distutils to try to add the right compiler/linker flags, without success. Allan
14.08.2015, 20:45, Allan Haldane kirjoitti: [clip]
Related to this, does anyone know how to debug numpy in gdb with proper symbols/source lines, like I can do with other C extensions? I've tried modifying numpy distutils to try to add the right compiler/linker flags, without success.
runtests.py --help gdb --args python runtests.py -g --python script.py grep env runtests.py
On 08/14/2015 01:52 PM, Pauli Virtanen wrote:
14.08.2015, 20:45, Allan Haldane kirjoitti: [clip]
Related to this, does anyone know how to debug numpy in gdb with proper symbols/source lines, like I can do with other C extensions? I've tried modifying numpy distutils to try to add the right compiler/linker flags, without success.
runtests.py --help
gdb --args python runtests.py -g --python script.py
grep env runtests.py
Oh! Thank you, I missed that.
On Aug 14, 2015 09:16, "Chris Barker" <chris.barker@noaa.gov> wrote:
On Thu, Aug 13, 2015 at 11:25 AM, Stefan van der Walt <
stefanv@berkeley.edu> wrote:
(for example "python setup.py develop", although suggested by setup.py itself, claims that "develop" is not a command).
develop is a command provided by setuptools, not distutils itself.
I find it absolutely invaluable -- it is THE way to go when actively working on any package under development.
if numpy doesn't currently use setuptools, it probably should (though maybe it's gets messy with numpy's distutils extensions...)
Regarding using setuptools by default, one problem is that it actually acts rather differently from distutils by default. See https://bitbucket.org/pypa/setuptools/issues/371/setuptools-and-state-of-pep...
Nowadays, you can use
pip install -e .
pip "injects" setuptools into the mix -- so this may be develope mode with a different name. but yes, a fine option for a package that doesn't use setuptools out of the box.
The version of setuptools that pip injects is also monkeypatched by pip to fix some of setuptools more obnoxious defaults. (The ones described in that bug report.) Using pip is also is the only way to reliably install all the right metadata needed to avoid problems later -- in particular pip will record the information needed to do uninstall/upgrades correctly, which neither distutils nor setuptools will do if you run setup.py directly. Basically this means running 'setup.py install' is always broken, for all projects and no matter how setup.py is written, and you should always run a pip command instead, even when building from the source tree. This is true for every python package, though, not just numpy. So setuptools doesn't provide much that's compelling for us... I believe if you really want it, though, you can run numpy's setupegg.py, which is the same as setup.py but using setuptools. Or something like that? I share Benjamin's doubts about the whole 'develop' approach, though, however accessed. For pure python packages, just importing from the source tree directly works fine and is way less error prone. For non-pure packages, I don't trust develop much anyway... build_ext --inplace can work nicely, or for numpy in particular runtests solves all my problems. (Though even then I still sometimes need to nuke the build directory or run clean manually.) -n
On Fri, Aug 14, 2015 at 10:08 AM, Benjamin Root <ben.root@ou.edu> wrote:
I used to be a huge advocate for the "develop" mode, but not anymore. I have run into way too many Heisenbugs that would clear up if I nuked my source tree and re-clone.
well, you do need to remember to clean out once in a while, when somethign weird is happening... But I prefer that to the other options, which are: * re-builda nd re-install with every frikin' change * do sys.path manipulations, which is ugly,, error prone, and has the same problems as develop mode anyway * rely on relative imports for all your tests and the like -- error prone and ugly -- oh, and you still have the problems above... I should also note that there is currently an open issue with "pip install
-e" and namespace packages.
yeah, I actually gave up on namespace packages due to them not working right with develop mode. (I'm not sure if -e and develop mode are exactly the same or not...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I think it's clear that develop/-e does not work well together with namespace packages. As noted on the relevant matplotlib issue https://github.com/matplotlib/matplotlib/issues/4907 I think the issue with namespace packages is essentially this well known one https://github.com/pypa/pip/issues/3 which I think I agree with Chris is enough to drop namespace packages if possible.
From the output of pip install -e I would say that it clear that it calls develop.
Since pip install -e and and pip install uses fundamentally different ways of manage namespace packages they can't work together. In the case of matplotlib issue #4907 basemap is probably installed into the namespace with pip install while matplotlib is installed with pip install -e which clearly triggers the issue in https://github.com/pypa/pip/issues/3 best Jens fre. 14. aug. 2015 kl. 21.21 skrev Chris Barker <chris.barker@noaa.gov>:
On Fri, Aug 14, 2015 at 10:08 AM, Benjamin Root <ben.root@ou.edu> wrote:
I used to be a huge advocate for the "develop" mode, but not anymore. I have run into way too many Heisenbugs that would clear up if I nuked my source tree and re-clone.
well, you do need to remember to clean out once in a while, when somethign weird is happening...
But I prefer that to the other options, which are:
* re-builda nd re-install with every frikin' change
* do sys.path manipulations, which is ugly,, error prone, and has the same problems as develop mode anyway
* rely on relative imports for all your tests and the like -- error prone and ugly -- oh, and you still have the problems above...
I should also note that there is currently an open issue with "pip install
-e" and namespace packages.
yeah, I actually gave up on namespace packages due to them not working right with develop mode.
(I'm not sure if -e and develop mode are exactly the same or not...)
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Aug 14, 2015 at 10:15 AM, Stefan van der Walt <stefanv@berkeley.edu> wrote:
Perhaps mpl_toolkits should think of becoming mpl_3d, mpl_basemaps, etc.?
namespace packages are a fine idea, but the implementation(s) are just one big kludge... I think so, but we're getting off-topic here. numpy doesn't use namespace packages, so develop mode works there. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
15.08.2015, 01:44, Chris Barker kirjoitti: [clip]
numpy doesn't use namespace packages, so develop mode works there.
The develop mode is mainly useful with a virtualenv. Otherwise, you install work-in-progress development version into your ~/.local which then breaks everything else. In addition to this, "python setupegg.py develop --uninstall" says "Note: you must uninstall or replace scripts manually!", and since the scripts end up with dev version requirement hardcoded, and you have to delete the scripts manually. Virtualenvs are annoying to manage, and at least for me personally it's easier to just deal with pythonpath, especially as runtests.py manages that. Anyway, TIMTOWTDI
On Sat, Aug 15, 2015 at 1:08 AM, Pauli Virtanen <pav@iki.fi> wrote:
15.08.2015, 01:44, Chris Barker kirjoitti: [clip]
numpy doesn't use namespace packages, so develop mode works there.
The develop mode is mainly useful with a virtualenv.
Otherwise, you install work-in-progress development version into your ~/.local which then breaks everything else. In addition to this, "python setupegg.py develop --uninstall" says "Note: you must uninstall or replace scripts manually!", and since the scripts end up with dev version requirement hardcoded, and you have to delete the scripts manually.
Virtualenvs are annoying to manage, and at least for me personally it's easier to just deal with pythonpath, especially as runtests.py manages that.
I completely agree. Virtualenv/pip/setuptools all too many issues and corner cases where things don't quite work for development purposes. Using runtests.py is the most reliable approach for working on numpy (or just use an in-place build + pythonpath management if you prefer). To get back to the original question of Anne (as well as the gdb one): most of what was said and recommended in this thread is fairly well documented in https://github.com/numpy/numpy/blob/master/doc/source/dev/development_enviro... Unfortunately it doesn't yet show up in http://docs.scipy.org/doc/numpy-dev/dev/index.html because that hasn't been updated in a while. If someone who knows how to do that could push an update, that would be great. Cheers, Ralf
participants (11)
-
Allan Haldane -
Anne Archibald -
Benjamin Root -
Charles R Harris -
Chris Barker -
Jens Nielsen -
Nathaniel Smith -
Pauli Virtanen -
Ralf Gommers -
Sebastian Berg -
Stefan van der Walt