Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
(I.e., pip install isn't just for packages on pypi -- you can also pass it the path to an arbitrary source directory or the URL of a source tarball and it will do its thing. In this case "install ." means "install the project in the current directory".)
These don't quite have identical results -- the main difference is that the latter makes sure that proper metadata gets installed so that later on it will be possible to upgrade or uninstall correctly. If you call setup.py directly, and then later you try to upgrade your package, then it's entirely possible to end up with a mixture of old and new versions of the package installed in your PYTHONPATH. (One common effect is in numpy's case is that we get people sending us mysterious bug reports about failing tests in files don't even exist (!) -- because nose is finding tests in files from one version of numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of weird corner cases in distutils/setuptools. E.g., if setup.py uses plain distutils, then it turns out this will mangle numpy version numbers in ways that cause weird horribleness -- see [1] for a bug report of the form "matplotlib doesn't build anymore" which turned out to be because of using 'setup.py install' to install numpy. OTOH if setup.py uses setuptools then you get different weirdnesses, like you can easily end up with multiple versions of the same library installed simultaneously.
And finally, an advantage of getting used to using 'pip install .' now is that you'll be prepared for the glorious future when we kill distutils and get rid of setup.py entirely in favor of something less terrible [2].
So a proposal that came out of the discussion in [1] is that we modify numpy's setup.py now so that if you try running
python setup.py install
you get
Error: Calling 'setup.py install' directly is NOT SUPPORTED! Instead, do:
pip install .
Alternatively, if you want to proceed at your own risk, you can try 'setup.py install --force-raw-setup.py' For more information see http://...
(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
This would hopefully cut down on the amount of time everyone spends trying to track down these stupid weird bugs, but it will also require some adjustment in people's workflows, so... objections? concerns?
-n
[1] https://github.com/numpy/numpy/issues/6551 [2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith njs@pobox.com wrote: [...]
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
On further investigation, it looks like the simplest approach to doing this would actually treat easy_install and setup_requires= the same way as they treat pip, i.e., they would all be allowed. (I was misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely upgraded, so I guess allowing this is okay :-).
-n
On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith njs@pobox.com wrote:
On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith njs@pobox.com wrote: [...]
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
On further investigation, it looks like the simplest approach to doing this would actually treat easy_install and setup_requires= the same way as they treat pip, i.e., they would all be allowed. (I was misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which ensures that setup_requires uses pip instead of easy_install. So we can not only keep setup-requires working, but make it work significantly better.
So if/when we accept the proposal in this thread, I'm thinking we should make a bunch of changes at once: - always use setuptools (this is a new dependency) - error on ``python setup.py install`` - add the setup-requires trick - error on ``python setup.py clean`` (saying "use `git clean -xdf` (or -Xdf ...) instead") - change ``python setup.py --help`` to first show numpy-specific stuff before setuptools help info - update all our install docs
And when "pip upgrade" is released (should be soon, see https://github.com/pypa/pip/pull/3194), officially change our mind and recommend the use of install_requires/setup_requires to packages depending on numpy.
Ralf
On Tue, Oct 27, 2015 at 12:19 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith njs@pobox.com wrote:
On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith njs@pobox.com wrote: [...]
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
On further investigation, it looks like the simplest approach to doing this would actually treat easy_install and setup_requires= the same way as they treat pip, i.e., they would all be allowed. (I was misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which ensures that setup_requires uses pip instead of easy_install. So we can not only keep setup-requires working, but make it work significantly better.
IIUC this is not something that we (= numpy) could use ourselves, but instead something that everyone who does setup_requires=["numpy"] would have to set up in their individual projects?
-n
On Tue, Oct 27, 2015 at 8:28 AM, Nathaniel Smith njs@pobox.com wrote:
On Tue, Oct 27, 2015 at 12:19 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith njs@pobox.com wrote:
On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith njs@pobox.com wrote: [...]
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
On further investigation, it looks like the simplest approach to doing this would actually treat easy_install and setup_requires= the same way as they treat pip, i.e., they would all be allowed. (I was misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which
ensures
that setup_requires uses pip instead of easy_install. So we can not only keep setup-requires working, but make it work significantly better.
IIUC this is not something that we (= numpy) could use ourselves, but instead something that everyone who does setup_requires=["numpy"] would have to set up in their individual projects?
Right. I was thinking about using it in scipy. Ah well, I'm sure we can manage to not break ``setup_requires=numpy`` in some way.
Ralf
On Tue, Oct 27, 2015 at 3:32 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 8:28 AM, Nathaniel Smith njs@pobox.com wrote:
On Tue, Oct 27, 2015 at 12:19 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith njs@pobox.com wrote:
On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith njs@pobox.com
wrote:
[...]
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
On further investigation, it looks like the simplest approach to doing this would actually treat easy_install and setup_requires= the same way as they treat pip, i.e., they would all be allowed. (I was misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which
ensures
that setup_requires uses pip instead of easy_install. So we can not only keep setup-requires working, but make it work significantly better.
IIUC this is not something that we (= numpy) could use ourselves, but instead something that everyone who does setup_requires=["numpy"] would have to set up in their individual projects?
Right. I was thinking about using it in scipy. Ah well, I'm sure we can manage to not break ``setup_requires=numpy`` in some way.
What's the equivalent of python setup.py build_ext --inplace
brief google search (I didn't follow up on those)
https://github.com/pypa/pip/issues/1887 https://github.com/pypa/pip/issues/18
Given that I rely completely on binary distributions for numpy and scipy, I won't be affected.
(I'm still allergic to pip and will switch only several years after everybody else.)
Josef
Ralf
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Oct 27, 2015 6:08 AM, josef.pktd@gmail.com wrote:
[...]
What's the equivalent of python setup.py build_ext --inplace
It's python setup.py build_ext --inplace
;-)
There's also no replacement for setup.py sdist, or setup.py upload (which is broken and should never be used), or setup.py clean (which is also broken and should never be used in numpy's case). pip is a better package installer than raw distutils or setuptools, for non-installation-related tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)
-n
On Tue, Oct 27, 2015 at 10:59 AM, Nathaniel Smith njs@pobox.com wrote:
On Oct 27, 2015 6:08 AM, josef.pktd@gmail.com wrote:
[...]
What's the equivalent of python setup.py build_ext --inplace
It's python setup.py build_ext --inplace
;-)
Ok, Sorry, I read now the small print and the issue.
Sounds reasonable, given we can `force` our way out.
(If the reason to run to pip is a misspelled dev version number, then it looks like a hammer to me.)
Josef
There's also no replacement for setup.py sdist, or setup.py upload (which is broken and should never be used), or setup.py clean (which is also broken and should never be used in numpy's case). pip is a better package installer than raw distutils or setuptools, for non-installation-related tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)
-n
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Would this happen at the level of numpy's setup.py script or would it be implemented in numpy.distutils? I'm asking as the developer of a package that uses numpy.distutils to manage C extensions.
On Tue, Oct 27, 2015 at 10:28 AM, josef.pktd@gmail.com wrote:
On Tue, Oct 27, 2015 at 10:59 AM, Nathaniel Smith njs@pobox.com wrote:
On Oct 27, 2015 6:08 AM, josef.pktd@gmail.com wrote:
[...]
What's the equivalent of python setup.py build_ext --inplace
It's python setup.py build_ext --inplace
;-)
Ok, Sorry, I read now the small print and the issue.
Sounds reasonable, given we can `force` our way out.
(If the reason to run to pip is a misspelled dev version number, then it looks like a hammer to me.)
Josef
There's also no replacement for setup.py sdist, or setup.py upload (which is broken and should never be used), or setup.py clean (which is also broken and should never be used in numpy's case). pip is a better package installer than raw distutils or setuptools, for non-installation-related tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)
-n
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Oct 27, 2015 8:34 AM, "Nathan Goldbaum" nathan12343@gmail.com wrote:
Would this happen at the level of numpy's setup.py script or would it be
implemented in numpy.distutils? I'm asking as the developer of a package that uses numpy.distutils to manage C extensions.
NumPy's setup.py, no effect on numpy.distutils users. Unless you also get fed up and implement the same thing, of course ;-).
-n
On Tue, Oct 27, 2015 at 4:28 PM, josef.pktd@gmail.com wrote:
On Tue, Oct 27, 2015 at 10:59 AM, Nathaniel Smith njs@pobox.com wrote:
On Oct 27, 2015 6:08 AM, josef.pktd@gmail.com wrote:
[...]
What's the equivalent of python setup.py build_ext --inplace
It's python setup.py build_ext --inplace
;-)
Ok, Sorry, I read now the small print and the issue.
Sounds reasonable, given we can `force` our way out.
(If the reason to run to pip is a misspelled dev version number, then it looks like a hammer to me.)
That's not the reason. A main reason is that we want reliable uninstall of numpy, as explained in the initial post. We also want to avoid as many other broken parts of setuptools/easy_install as possible.
Ralf
On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
Updating this list for comments made after I sent it and now that I've looked in more detail at what the less common commands do:
So if/when we accept the proposal in this thread, I'm thinking we should make a bunch of changes at once:
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s` instead") - error on ``python setup.py upload_docs`` - error on ``python setup.py easy_install`` (I'm not joking, that exists) - error on ``python setup.py test`` (saying "use `python runtests.py` instead") - remove setupegg.py
Ralf
And when "pip upgrade" is released (should be soon, see
https://github.com/pypa/pip/pull/3194), officially change our mind and recommend the use of install_requires/setup_requires to packages depending on numpy.
Can someone here who understands more about distribution maybe write a blog post detailing:
- why these setup.py commands are bad
- which alternative corresponds to each command and why it's better
- where to find information about this
For example, I had never heard of "twine", and parenthetical statements such as "setup.py upload (which is broken and should never be used)" are useless to those who don't know this and useless to those who do.
I understand that this is an "internal" discussion, but it's nice if those following to learn can get quick pointers. Since there is a *ton* of material online telling us *to use* python setup.py install, all the time, it would be extremely helpful for the community if discussions such as this one helped to bubble up the Right Way of doing Python packaging and distribution.
Thanks,
Juan.
On Wed, Oct 28, 2015 at 9:16 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers ralf.gommers@gmail.com wrote: Updating this list for comments made after I sent it and now that I've looked in more detail at what the less common commands do:
So if/when we accept the proposal in this thread, I'm thinking we should make a bunch of changes at once:
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py
Ralf And when "pip upgrade" is released (should be soon, see
https://github.com/pypa/pip/pull/3194), officially change our mind and recommend the use of install_requires/setup_requires to packages depending on numpy.
On Tue, Oct 27, 2015 at 11:35 PM, Juan Nunez-Iglesias jni.soma@gmail.com wrote:
Can someone here who understands more about distribution maybe write a blog post detailing:
- why these setup.py commands are bad
- which alternative corresponds to each command and why it's better
- where to find information about this
Good question. Not that I have a blog, but I can try to write something a bit longer the coming weekend.
For example, I had never heard of "twine", and parenthetical statements such as "setup.py upload (which is broken and should never be used)" are useless to those who don't know this and useless to those who do.
IIRC `setup.py upload` sends passwords over plain http. I've also seen it do weird things like change one's own PyPi rights from maintainer to owner.
The most comprehensive overview of all this stuff is https://packaging.python.org/en/latest/, which starts with tool recommendations. Twine is one of the first things mentioned.
Ralf
I understand that this is an "internal" discussion, but it's nice if those following to learn can get quick pointers. Since there is a *ton* of material online telling us *to use* python setup.py install, all the time, it would be extremely helpful for the community if discussions such as this one helped to bubble up the Right Way of doing Python packaging and distribution.
Thanks,
Juan.
On Wed, Oct 28, 2015 at 9:16 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
Updating this list for comments made after I sent it and now that I've looked in more detail at what the less common commands do:
So if/when we accept the proposal in this thread, I'm thinking we should make a bunch of changes at once:
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py
Ralf
And when "pip upgrade" is released (should be soon, see
https://github.com/pypa/pip/pull/3194), officially change our mind and recommend the use of install_requires/setup_requires to packages depending on numpy.
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Thanks Ralf! The pointer to Python Packaging User Guide is already gold! (But a wider discussion e.g. in the NumPy repo, mirroring the docstring conventions, would also be good!)
On Wed, Oct 28, 2015 at 10:02 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 11:35 PM, Juan Nunez-Iglesias jni.soma@gmail.com wrote:
Can someone here who understands more about distribution maybe write a blog post detailing:
- why these setup.py commands are bad
- which alternative corresponds to each command and why it's better
- where to find information about this
Good question. Not that I have a blog, but I can try to write something a bit longer the coming weekend.
For example, I had never heard of "twine", and parenthetical statements such as "setup.py upload (which is broken and should never be used)" are useless to those who don't know this and useless to those who do.
IIRC `setup.py upload` sends passwords over plain http. I've also seen it do weird things like change one's own PyPi rights from maintainer to owner. The most comprehensive overview of all this stuff is https://packaging.python.org/en/latest/, which starts with tool recommendations. Twine is one of the first things mentioned. Ralf
I understand that this is an "internal" discussion, but it's nice if those following to learn can get quick pointers. Since there is a *ton* of material online telling us *to use* python setup.py install, all the time, it would be extremely helpful for the community if discussions such as this one helped to bubble up the Right Way of doing Python packaging and distribution.
Thanks,
Juan.
On Wed, Oct 28, 2015 at 9:16 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
Updating this list for comments made after I sent it and now that I've looked in more detail at what the less common commands do:
So if/when we accept the proposal in this thread, I'm thinking we should make a bunch of changes at once:
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py
Ralf
And when "pip upgrade" is released (should be soon, see
https://github.com/pypa/pip/pull/3194), officially change our mind and recommend the use of install_requires/setup_requires to packages depending on numpy.
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, 27 Oct 2015 15:35:50 -0700 (PDT) "Juan Nunez-Iglesias" jni.soma@gmail.com wrote:
Can someone here who understands more about distribution maybe write a blog post detailing:
Hi,
Olivier Grisel from sklearn gave a very good talk on this topic at PyCon, earlier this year: http://www.pyvideo.org/video/3473/build-and-test-wheel-packages-on-linux-osx...
Very instructive.
Thanks, Jerome! I’ve added it to my to-watch list. It sounds really useful!
Juan.
On Wed, Oct 28, 2015 at 6:36 PM, Jerome Kieffer Jerome.Kieffer@esrf.fr wrote:
On Tue, 27 Oct 2015 15:35:50 -0700 (PDT) "Juan Nunez-Iglesias" jni.soma@gmail.com wrote:
Can someone here who understands more about distribution maybe write a blog post detailing:
Hi, Olivier Grisel from sklearn gave a very good talk on this topic at PyCon, earlier this year: http://www.pyvideo.org/video/3473/build-and-test-wheel-packages-on-linux-osx... Very instructive. -- Jérôme Kieffer tel +33 476 882 445 _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Mon, Oct 26, 2015 at 10:31 PM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
(I.e., pip install isn't just for packages on pypi -- you can also pass it the path to an arbitrary source directory or the URL of a source tarball and it will do its thing. In this case "install ." means "install the project in the current directory".)
These don't quite have identical results -- the main difference is that the latter makes sure that proper metadata gets installed so that later on it will be possible to upgrade or uninstall correctly. If you call setup.py directly, and then later you try to upgrade your package, then it's entirely possible to end up with a mixture of old and new versions of the package installed in your PYTHONPATH. (One common effect is in numpy's case is that we get people sending us mysterious bug reports about failing tests in files don't even exist (!) -- because nose is finding tests in files from one version of numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of weird corner cases in distutils/setuptools. E.g., if setup.py uses plain distutils, then it turns out this will mangle numpy version numbers in ways that cause weird horribleness -- see [1] for a bug report of the form "matplotlib doesn't build anymore" which turned out to be because of using 'setup.py install' to install numpy. OTOH if setup.py uses setuptools then you get different weirdnesses, like you can easily end up with multiple versions of the same library installed simultaneously.
And finally, an advantage of getting used to using 'pip install .' now is that you'll be prepared for the glorious future when we kill distutils and get rid of setup.py entirely in favor of something less terrible [2].
So a proposal that came out of the discussion in [1] is that we modify numpy's setup.py now so that if you try running
python setup.py installyou get
Error: Calling 'setup.py install' directly is NOT SUPPORTED! Instead, do: pip install . Alternatively, if you want to proceed at your own risk, you can try 'setup.py install --force-raw-setup.py' For more information see http://...(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
This would hopefully cut down on the amount of time everyone spends trying to track down these stupid weird bugs, but it will also require some adjustment in people's workflows, so... objections? concerns?
I gave it a shot the other day. Pip keeps a record of the path to the repo and in order to cleanup I needed to search out the file and delete the repo path. There is probably a better way to do that, but it didn't strike me as less troublesome than ` python setup.py install --local`.
Chuck
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris charlesr.harris@gmail.com wrote:
[...]
I gave it a shot the other day. Pip keeps a record of the path to the repo and in order to cleanup I needed to search out the file and delete the repo path. There is probably a better way to do that, but it didn't strike me as less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create? What does `setup.py install --local` do? (it doesn't seem to be mentioned in `setup.py install --help`.)
-n
On Tue, Oct 27, 2015 at 12:08 AM, Nathaniel Smith njs@pobox.com wrote:
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris charlesr.harris@gmail.com wrote:
[...]
I gave it a shot the other day. Pip keeps a record of the path to the
repo
and in order to cleanup I needed to search out the file and delete the
repo
path. There is probably a better way to do that, but it didn't strike me
as
less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create? What does `setup.py install --local` do? (it doesn't seem to be mentioned in `setup.py install --help`.)
`pip install --user -e . `. However, `pip install --user .` seems to work fine. The pip documentation isn't the best.
Yeah, `--user` not `--local`. It's getting late...
Chuck
Is there a pip equivalent of "python setup.py develop"?
On Tue, Oct 27, 2015 at 5:33 PM Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, Oct 27, 2015 at 12:08 AM, Nathaniel Smith njs@pobox.com wrote:
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris charlesr.harris@gmail.com wrote:
[...]
I gave it a shot the other day. Pip keeps a record of the path to the
repo
and in order to cleanup I needed to search out the file and delete the
repo
path. There is probably a better way to do that, but it didn't strike
me as
less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create? What does `setup.py install --local` do? (it doesn't seem to be mentioned in `setup.py install --help`.)
`pip install --user -e . `. However, `pip install --user .` seems to work fine. The pip documentation isn't the best.
Yeah, `--user` not `--local`. It's getting late...
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Mon, Oct 26, 2015 at 11:53 PM, Juan Nunez-Iglesias jni.soma@gmail.com wrote:
Is there a pip equivalent of "python setup.py develop"?
Kinda answered this already when replying to Chuck, but: yes, it's `pip install -e <path or URL or package name>` (the -e is short for --editable), not that you would need it necessarily because `setup.py develop` would still be legal.
-n
On Mon, Oct 26, 2015 at 11:33 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, Oct 27, 2015 at 12:08 AM, Nathaniel Smith njs@pobox.com wrote:
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris charlesr.harris@gmail.com wrote:
[...]
I gave it a shot the other day. Pip keeps a record of the path to the repo and in order to cleanup I needed to search out the file and delete the repo path. There is probably a better way to do that, but it didn't strike me as less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create? What does `setup.py install --local` do? (it doesn't seem to be mentioned in `setup.py install --help`.)
`pip install --user -e . `. However, `pip install --user .` seems to work fine. The pip documentation isn't the best.
Yeah, `--user` not `--local`. It's getting late...
Ah. I think if you want to undo a `pip install --user -e .` you can just do `pip uninstall numpy`? But in any case the equivalent of 'pip install -e' is 'setup.py develop', and the proposal is only to disable `setup.py install`. So if you prefer `setup.py develop` then you could still use it.
(IIUC `pip install -e` is a *very* thin wrapper around `setup.py develop` -- normally pip takes over the job of actually installing files, so pip gets to interpret options like --user and figure out what it means for where it puts the files, but in the case of `install -e` it looks like it just calls `setup.py develop` directly, so if --user doesn't work it's probably because setup.py develop doesn't know about --user?)
-n
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that pip is bad & broken, whereas python setup.py install does what I expect (even if it's a mess internally). In particular, when maintaining a distribution of Python packages, you try to have some well-defined, reproducible build from source tarballs and then you find that pip is going off and downloading stuff under the radar without being asked (etc.). Stopping that can be a pain & I always groan whenever some package insists on using pip. Maybe I don't understand it well enough but in this role its dependency handling is an unnecessary complication with no purpose. Just a comment that not every installation is someone trying to get numpy on their laptop...
Cheers,
James.
I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned Conda yet.
Is there any particular reason for not using it?
On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner jehturner@gmail.com wrote:
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that pip is bad & broken, whereas python setup.py install does what I expect (even if it's a mess internally). In particular, when maintaining a distribution of Python packages, you try to have some well-defined, reproducible build from source tarballs and then you find that pip is going off and downloading stuff under the radar without being asked (etc.). Stopping that can be a pain & I always groan whenever some package insists on using pip. Maybe I don't understand it well enough but in this role its dependency handling is an unnecessary complication with no purpose. Just a comment that not every installation is someone trying to get numpy on their laptop...
Cheers,
James.
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 27, 2015 at 2:31 PM, Edison Gustavo Muenz < edisongustavo@gmail.com> wrote:
I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned Conda yet.
Conda is a binary distribution system, whereas we are talking about installing from sources. You will need a way to install things when building a conda package in any case
David
Is there any particular reason for not using it?
On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner jehturner@gmail.com wrote:
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that pip is bad & broken, whereas python setup.py install does what I expect (even if it's a mess internally). In particular, when maintaining a distribution of Python packages, you try to have some well-defined, reproducible build from source tarballs and then you find that pip is going off and downloading stuff under the radar without being asked (etc.). Stopping that can be a pain & I always groan whenever some package insists on using pip. Maybe I don't understand it well enough but in this role its dependency handling is an unnecessary complication with no purpose. Just a comment that not every installation is someone trying to get numpy on their laptop...
Cheers,
James.
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Conda is for binary installs and largely targeted for end-users. This topic pertains to source installs, and is mostly relevant to developers, testers, and those who like to live on the bleeding edge of a particular project.
On Tue, Oct 27, 2015 at 10:31 AM, Edison Gustavo Muenz < edisongustavo@gmail.com> wrote:
I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned Conda yet.
Is there any particular reason for not using it?
On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner jehturner@gmail.com wrote:
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that pip is bad & broken, whereas python setup.py install does what I expect (even if it's a mess internally). In particular, when maintaining a distribution of Python packages, you try to have some well-defined, reproducible build from source tarballs and then you find that pip is going off and downloading stuff under the radar without being asked (etc.). Stopping that can be a pain & I always groan whenever some package insists on using pip. Maybe I don't understand it well enough but in this role its dependency handling is an unnecessary complication with no purpose. Just a comment that not every installation is someone trying to get numpy on their laptop...
Cheers,
James.
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Oct 27, 2015 6:48 AM, "James E.H. Turner" jehturner@gmail.com wrote:
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that pip is bad & broken, whereas python setup.py install does what I expect (even if it's a mess internally).
Unfortunately this is only true if what you expect is for packages to be installed in subtly corrupted ways, as described in the original email. Sorry to be the bearer of bad tidings :-/
In particular, when maintaining a distribution of Python packages, you try to have some well-defined, reproducible build from source tarballs and then you find that pip is going off and downloading stuff under the radar without being asked (etc.). Stopping that can be a pain & I always groan whenever some package insists on using pip. Maybe I don't understand it well enough but in this role its dependency handling is an unnecessary complication with no purpose.
There are two cases where a 'pip install' run might go off and start downloading packages without asking you:
- if the project is using setuptools with setup_requires=..., then the setup.py itself will go off and start downloading things without asking. This has nothing to do with pip. The way Debian prevents this is that they always define an intentionally invalid http_proxy environment variable before building any python package.
- if the project has declared that they do not work without some other package installed via install_requires=... For this case, if you really know what you're doing and you intentionally want to install a non-functional configuration (which yeah, a package build tool might indeed want to do), then just add --no-deps to the pip install command line. Maybe add --no-index and/or the magic http_proxy setting if you want to be extra sure.
Just a comment that not every installation is someone trying to get numpy on their laptop...
Sure, we're well aware of the importance of downstream packagers -- part of the point of having this email thread is to smoke out such non-trivial use cases. (And note that worst case if you decide that you'd rather take your chances with setup.py install, then that's why the proposal includes an escape hatch of passing a special --force switch.)
But unless you're somehow planning to disable pip entirely in your distribution, so that end users have to get upgrades through your tools rather than using pip, then you do probably want to think about how to provide accurate pip-style metadata. (And even then it doesn't hurt.)
-n
Interestingly, conda actually does "setup.py install" in the recipe for numpy: https://github.com/conda/conda-recipes/blob/master/numpy-openblas/build.sh
I'm not sure if this is the one they use to build the anaconda package, I think they have internal versions of most of the recipes on conda-recipes.
On Tue, Oct 27, 2015 at 10:18 AM, Nathaniel Smith njs@pobox.com wrote:
On Oct 27, 2015 6:48 AM, "James E.H. Turner" jehturner@gmail.com wrote:
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that pip is bad & broken, whereas python setup.py install does what I expect (even if it's a mess internally).
Unfortunately this is only true if what you expect is for packages to be installed in subtly corrupted ways, as described in the original email. Sorry to be the bearer of bad tidings :-/
In particular, when maintaining a distribution of Python packages, you try to have some well-defined, reproducible build from source tarballs and then you find that pip is going off and downloading stuff under the radar without being asked (etc.). Stopping that can be a pain & I always groan whenever some package insists on using pip. Maybe I don't understand it well enough but in this role its dependency handling is an unnecessary complication with no purpose.
There are two cases where a 'pip install' run might go off and start downloading packages without asking you:
- if the project is using setuptools with setup_requires=..., then the
setup.py itself will go off and start downloading things without asking. This has nothing to do with pip. The way Debian prevents this is that they always define an intentionally invalid http_proxy environment variable before building any python package.
- if the project has declared that they do not work without some other
package installed via install_requires=... For this case, if you really know what you're doing and you intentionally want to install a non-functional configuration (which yeah, a package build tool might indeed want to do), then just add --no-deps to the pip install command line. Maybe add --no-index and/or the magic http_proxy setting if you want to be extra sure.
Just a comment that not every installation is someone trying to get numpy on their laptop...
Sure, we're well aware of the importance of downstream packagers -- part of the point of having this email thread is to smoke out such non-trivial use cases. (And note that worst case if you decide that you'd rather take your chances with setup.py install, then that's why the proposal includes an escape hatch of passing a special --force switch.)
But unless you're somehow planning to disable pip entirely in your distribution, so that end users have to get upgrades through your tools rather than using pip, then you do probably want to think about how to provide accurate pip-style metadata. (And even then it doesn't hurt.)
-n
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 27, 2015 at 8:25 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Interestingly, conda actually does "setup.py install" in the recipe for numpy:
indeed -- many, many conda packages do setup.py install, whihc doesn't mean it's a good idea --personally, I'm trying hard to switch them all to:
pip install ./
:-)
Which reminds me, the conda skelaton command craes a pip install build.sh file -- I really need to submit a PR for that ...
There are two cases where a 'pip install' run might go off and start
downloading packages without asking you:
for my part, regular old setup.py isntall oftem goes off and istalls sutff too - using easy_install, which really sucks...
This is making me want a setuptools-lite again -- see the distutils SIG if you're curious.
-CHB
please, pretty please, do not disable setup.py install or at least keep providing a way for distribution (Debian in this case) to be able to build/install numpy in a temporary location for packaging reasons. pip is not the solution for us
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
(I.e., pip install isn't just for packages on pypi -- you can also pass it the path to an arbitrary source directory or the URL of a source tarball and it will do its thing. In this case "install ." means "install the project in the current directory".)
These don't quite have identical results -- the main difference is that the latter makes sure that proper metadata gets installed so that later on it will be possible to upgrade or uninstall correctly. If you call setup.py directly, and then later you try to upgrade your package, then it's entirely possible to end up with a mixture of old and new versions of the package installed in your PYTHONPATH. (One common effect is in numpy's case is that we get people sending us mysterious bug reports about failing tests in files don't even exist (!) -- because nose is finding tests in files from one version of numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of weird corner cases in distutils/setuptools. E.g., if setup.py uses plain distutils, then it turns out this will mangle numpy version numbers in ways that cause weird horribleness -- see [1] for a bug report of the form "matplotlib doesn't build anymore" which turned out to be because of using 'setup.py install' to install numpy. OTOH if setup.py uses setuptools then you get different weirdnesses, like you can easily end up with multiple versions of the same library installed simultaneously.
And finally, an advantage of getting used to using 'pip install .' now is that you'll be prepared for the glorious future when we kill distutils and get rid of setup.py entirely in favor of something less terrible [2].
So a proposal that came out of the discussion in [1] is that we modify numpy's setup.py now so that if you try running
python setup.py installyou get
Error: Calling 'setup.py install' directly is NOT SUPPORTED! Instead, do: pip install . Alternatively, if you want to proceed at your own risk, you can try 'setup.py install --force-raw-setup.py' For more information see http://...(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
This would hopefully cut down on the amount of time everyone spends trying to track down these stupid weird bugs, but it will also require some adjustment in people's workflows, so... objections? concerns?
-n
[1] https://github.com/numpy/numpy/issues/6551 [2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Thu, Oct 29, 2015 at 12:28 AM, Sandro Tosi morph@debian.org wrote:
please, pretty please, do not disable setup.py install or at least keep providing a way for distribution (Debian in this case) to be able to build/install numpy in a temporary location for packaging reasons. pip is not the solution for us
``python setup.py install --force`` would still work. Would that be OK then?
Ralf
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
(I.e., pip install isn't just for packages on pypi -- you can also pass it the path to an arbitrary source directory or the URL of a source tarball and it will do its thing. In this case "install ." means "install the project in the current directory".)
These don't quite have identical results -- the main difference is that the latter makes sure that proper metadata gets installed so that later on it will be possible to upgrade or uninstall correctly. If you call setup.py directly, and then later you try to upgrade your package, then it's entirely possible to end up with a mixture of old and new versions of the package installed in your PYTHONPATH. (One common effect is in numpy's case is that we get people sending us mysterious bug reports about failing tests in files don't even exist (!) -- because nose is finding tests in files from one version of numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of weird corner cases in distutils/setuptools. E.g., if setup.py uses plain distutils, then it turns out this will mangle numpy version numbers in ways that cause weird horribleness -- see [1] for a bug report of the form "matplotlib doesn't build anymore" which turned out to be because of using 'setup.py install' to install numpy. OTOH if setup.py uses setuptools then you get different weirdnesses, like you can easily end up with multiple versions of the same library installed simultaneously.
And finally, an advantage of getting used to using 'pip install .' now is that you'll be prepared for the glorious future when we kill distutils and get rid of setup.py entirely in favor of something less terrible [2].
So a proposal that came out of the discussion in [1] is that we modify numpy's setup.py now so that if you try running
python setup.py installyou get
Error: Calling 'setup.py install' directly is NOT SUPPORTED! Instead, do: pip install . Alternatively, if you want to proceed at your own risk, you can try 'setup.py install --force-raw-setup.py' For more information see http://...(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
This would hopefully cut down on the amount of time everyone spends trying to track down these stupid weird bugs, but it will also require some adjustment in people's workflows, so... objections? concerns?
-n
https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Oct 29, 2015 00:29, "Sandro Tosi" morph@debian.org wrote:
please, pretty please, do not disable setup.py install or at least keep providing a way for distribution (Debian in this case) to be able to build/install numpy in a temporary location for packaging reasons. pip is not the solution for us
What is wrong with "pip install --root" ?
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a lot of instructions involving `setup.py build` and `setup.py install`. See, for example, INSTALL.txt. Also see http://docs.scipy.org/doc/numpy/user/install.html#building-from-source So I guess it is not surprising that it is not well known.
Warren
(I.e., pip install isn't just for packages on pypi -- you can also pass it the path to an arbitrary source directory or the URL of a source tarball and it will do its thing. In this case "install ." means "install the project in the current directory".)
These don't quite have identical results -- the main difference is that the latter makes sure that proper metadata gets installed so that later on it will be possible to upgrade or uninstall correctly. If you call setup.py directly, and then later you try to upgrade your package, then it's entirely possible to end up with a mixture of old and new versions of the package installed in your PYTHONPATH. (One common effect is in numpy's case is that we get people sending us mysterious bug reports about failing tests in files don't even exist (!) -- because nose is finding tests in files from one version of numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of weird corner cases in distutils/setuptools. E.g., if setup.py uses plain distutils, then it turns out this will mangle numpy version numbers in ways that cause weird horribleness -- see [1] for a bug report of the form "matplotlib doesn't build anymore" which turned out to be because of using 'setup.py install' to install numpy. OTOH if setup.py uses setuptools then you get different weirdnesses, like you can easily end up with multiple versions of the same library installed simultaneously.
And finally, an advantage of getting used to using 'pip install .' now is that you'll be prepared for the glorious future when we kill distutils and get rid of setup.py entirely in favor of something less terrible [2].
So a proposal that came out of the discussion in [1] is that we modify numpy's setup.py now so that if you try running
python setup.py installyou get
Error: Calling 'setup.py install' directly is NOT SUPPORTED! Instead, do: pip install . Alternatively, if you want to proceed at your own risk, you can try 'setup.py install --force-raw-setup.py' For more information see http://...(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed.
This would hopefully cut down on the amount of time everyone spends trying to track down these stupid weird bugs, but it will also require some adjustment in people's workflows, so... objections? concerns?
-n
[1] https://github.com/numpy/numpy/issues/6551 [2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser < warren.weckesser@gmail.com> wrote:
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a lot of instructions involving `setup.py build` and `setup.py install`. See, for example, INSTALL.txt. Also see http://docs.scipy.org/doc/numpy/user/install.html#building-from-source So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many of them. There's duplicate info in INSTALL.txt and http://scipy.org/scipylib/building/index.html for example. We should probably just empty out INSTALL.txt and simply put a link in it to the html docs.
I've created an issue with a long todo list and a bunch of links: https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to go fix something:)
Ralf
On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser < warren.weckesser@gmail.com> wrote:
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a lot of instructions involving `setup.py build` and `setup.py install`. See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many of them. There's duplicate info in INSTALL.txt and http://scipy.org/scipylib/building/index.html for example. We should probably just empty out INSTALL.txt and simply put a link in it to the html docs.
I've created an issue with a long todo list and a bunch of links: https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered concerns (at least in my perception), so without more discussion I'd interpret the current status as "go ahead".
Ralf
On Sun, Nov 1, 2015 at 1:59 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser < warren.weckesser@gmail.com> wrote:
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a lot of instructions involving `setup.py build` and `setup.py install`. See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many of them. There's duplicate info in INSTALL.txt and http://scipy.org/scipylib/building/index.html for example. We should probably just empty out INSTALL.txt and simply put a link in it to the html docs.
I've created an issue with a long todo list and a bunch of links: https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered concerns (at least in my perception), so without more discussion I'd interpret the current status as "go ahead".
Hmm, after some more testing I'm going to have to bring up a few concerns myself:
1. ``pip install .`` still has a clear bug; it starts by copying everything (including .git/ !) to a tempdir with shutil, which is very slow. And the fix for that will go via ``setup.py sdist``, which is still slow.
2. ``pip install .`` silences build output, which may make sense for some usecases, but for numpy it just sits there for minutes with no output after printing "Running setup.py install for numpy". Users will think it hangs and Ctrl-C it. https://github.com/pypa/pip/issues/2732
3. ``pip install .`` refuses to upgrade an already installed development version. For released versions that makes sense, but if I'm in a git tree then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
I've sent a (incomplete) fix for the shutil thing ( https://github.com/pypa/pip/pull/3219) and will comment on some open issues on the pip tracker. But I'm thinking that for now we should go with some printed message first. Something like "please use ``pip install .`` if you want reliable uninstall behavior. See <somelink> for more details".
Pip has worked quite well for me in the past, but the above makes me thing it's not much of an improvement over use of setuptools.....
Ralf
On Sun, Nov 1, 2015 at 4:16 PM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Sun, Nov 1, 2015 at 1:59 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser < warren.weckesser@gmail.com> wrote:
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a lot of instructions involving `setup.py build` and `setup.py install`. See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many of them. There's duplicate info in INSTALL.txt and http://scipy.org/scipylib/building/index.html for example. We should probably just empty out INSTALL.txt and simply put a link in it to the html docs.
I've created an issue with a long todo list and a bunch of links: https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered concerns (at least in my perception), so without more discussion I'd interpret the current status as "go ahead".
Hmm, after some more testing I'm going to have to bring up a few concerns myself:
- ``pip install .`` still has a clear bug; it starts by copying
everything (including .git/ !) to a tempdir with shutil, which is very slow. And the fix for that will go via ``setup.py sdist``, which is still slow.
- ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after printing "Running setup.py install for numpy". Users will think it hangs and Ctrl-C it. https://github.com/pypa/pip/issues/2732
- ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
I've sent a (incomplete) fix for the shutil thing ( https://github.com/pypa/pip/pull/3219) and will comment on some open issues on the pip tracker. But I'm thinking that for now we should go with some printed message first. Something like "please use ``pip install .`` if you want reliable uninstall behavior. See <somelink> for more details".
Pip has worked quite well for me in the past, but the above makes me thing it's not much of an improvement over use of setuptools.....
Which version of pip?
Chuck
On Mon, Nov 2, 2015 at 1:12 AM, Charles R Harris charlesr.harris@gmail.com wrote:
On Sun, Nov 1, 2015 at 4:16 PM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Sun, Nov 1, 2015 at 1:59 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers ralf.gommers@gmail.com wrote:
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser < warren.weckesser@gmail.com> wrote:
On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith njs@pobox.com wrote:
Hi all,
Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a lot of instructions involving `setup.py build` and `setup.py install`. See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many of them. There's duplicate info in INSTALL.txt and http://scipy.org/scipylib/building/index.html for example. We should probably just empty out INSTALL.txt and simply put a link in it to the html docs.
I've created an issue with a long todo list and a bunch of links: https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered concerns (at least in my perception), so without more discussion I'd interpret the current status as "go ahead".
Hmm, after some more testing I'm going to have to bring up a few concerns myself:
- ``pip install .`` still has a clear bug; it starts by copying
everything (including .git/ !) to a tempdir with shutil, which is very slow. And the fix for that will go via ``setup.py sdist``, which is still slow.
- ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after printing "Running setup.py install for numpy". Users will think it hangs and Ctrl-C it. https://github.com/pypa/pip/issues/2732
- ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
I've sent a (incomplete) fix for the shutil thing ( https://github.com/pypa/pip/pull/3219) and will comment on some open issues on the pip tracker. But I'm thinking that for now we should go with some printed message first. Something like "please use ``pip install .`` if you want reliable uninstall behavior. See <somelink> for more details".
Pip has worked quite well for me in the past, but the above makes me thing it's not much of an improvement over use of setuptools.....
Which version of pip?
Latest master (it's 'develop' branch). Recent released versions will be the same, because there are open issues for these things.
Ralf
[Adding distutils-sig to the CC as a heads-up. The context is that numpy is looking at deprecating the use of 'python setup.py install' and enforcing the use of 'pip install .' instead, and running into some issues that will probably need to be addressed if 'pip install .' is going to become the standard interface to work with source trees.]
On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers ralf.gommers@gmail.com wrote: [...]
Hmm, after some more testing I'm going to have to bring up a few concerns myself:
- ``pip install .`` still has a clear bug; it starts by copying everything
(including .git/ !) to a tempdir with shutil, which is very slow. And the fix for that will go via ``setup.py sdist``, which is still slow.
Ugh. If 'pip (install/wheel) .' is supposed to become the standard way to build things, then it should probably build in-place by default. Working in a temp dir makes perfect sense for 'pip install <requirement>' or 'pip install <url>', but if the user supplies an actual named on-disk directory then presumably the user is expecting this directory to be used, and to be able to take advantage of incremental rebuilds etc., no?
- ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after printing "Running setup.py install for numpy". Users will think it hangs and Ctrl-C it. https://github.com/pypa/pip/issues/2732
I tend to agree with the commentary there that for end users this is different but no worse than the current situation where we spit out pages of "errors" that don't mean anything :-). I posted a suggestion on that bug that might help with the apparent hanging problem.
- ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
Ugh, this is clearly just a bug -- `pip install .` should always unconditionally install, IMO. (Did you file a bug yet?) At least the workaround is just 'pip uninstall numpy; pip install .', which is still better the running 'setup.py install' and having it blithely overwrite some files and not others.
The first and last issue seem like ones that will mostly only affect developers, who should mostly have the ability to deal with these weird issues (or just use setup.py install --force if that's what they prefer)? This still seems like a reasonable trade-off to me if it also has the effect of reducing the number of weird broken installs among our thousands-of-times-larger userbase.
-n
On Mon, Nov 2, 2015 at 5:57 PM, Nathaniel Smith njs@pobox.com wrote:
On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers ralf.gommers@gmail.com wrote:
- ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after printing "Running setup.py install for numpy". Users will think it hangs and Ctrl-C it. https://github.com/pypa/pip/issues/2732
I tend to agree with the commentary there that for end users this is different but no worse than the current situation where we spit out pages of "errors" that don't mean anything :-). I posted a suggestion on that bug that might help with the apparent hanging problem.
For the record, this is now fixed in pip's "develop" branch and should be in the next release. For commands like 'setup.py install', pip now displays a spinner that ticks over whenever the underlying process prints to stdout/stderr. So if the underlying process hangs, then the spinner will stop (it's not just lying to you), but normally it works nicely.
https://github.com/pypa/pip/pull/3224
-n
participants (15)
-
Benjamin Root -
Charles R Harris -
Chris Barker -
David Cournapeau -
Edison Gustavo Muenz -
James E.H. Turner -
Jerome Kieffer -
josef.pktd＠gmail.com -
Juan Nunez-Iglesias -
Nathan Goldbaum -
Nathaniel Smith -
Ralf Gommers -
Sandro Tosi -
Todd -
Warren Weckesser