specifying numpy as dependency in your project, install_requires
Hi, An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6'] in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that (https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list. The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason. This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py: try: import numpy except ImportError: raise ImportError("my_package requires numpy") or try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion) Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere? Ralf
On Sep 21, 2012, at 3:13 PM, Ralf Gommers wrote:
Hi,
An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6']
in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that (https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list.
The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason.
This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py:
try: import numpy except ImportError: raise ImportError("my_package requires numpy")
or
try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion)
Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere?
I agree. I would recommend against using install requires. -Travis
On Fri, Sep 21, 2012 at 4:19 PM, Travis Oliphant <travis@continuum.io>wrote:
On Sep 21, 2012, at 3:13 PM, Ralf Gommers wrote:
Hi,
An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6']
in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that ( https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list.
The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason.
This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py:
try: import numpy except ImportError: raise ImportError("my_package requires numpy")
or
try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion)
Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere?
I agree. I would recommend against using install requires.
-Travis
Why? I have personally never had an issue with this. The only way I could imagine that this wouldn't work is if numpy was installed via some other means and there wasn't an entry in the easy-install.pth (or whatever equivalent pip uses). If pip is having a problem detecting numpy, then that is a bug that needs fixing somewhere. As for packages getting updated unintentionally, easy_install and pip both require an argument to upgrade any existing packages (I think -U), so I am not sure how you are running into such a situation. I have found install_requires to be a powerful feature in my setup.py scripts, and I have seen no reason to discourage it. Perhaps I am the only one? Ben Root
Am Fr 21 Sep 2012 22:37:13 CEST schrieb Benjamin Root:
On Fri, Sep 21, 2012 at 4:19 PM, Travis Oliphant <travis@continuum.io <mailto:travis@continuum.io>> wrote:
On Sep 21, 2012, at 3:13 PM, Ralf Gommers wrote:
Hi,
An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6']
in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that (https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list.
The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason.
This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py:
try: import numpy except ImportError: raise ImportError("my_package requires numpy")
or
try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion)
Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere?
I agree. I would recommend against using install requires.
-Travis
Why? I have personally never had an issue with this. The only way I could imagine that this wouldn't work is if numpy was installed via some other means and there wasn't an entry in the easy-install.pth (or whatever equivalent pip uses). If pip is having a problem detecting numpy, then that is a bug that needs fixing somewhere.
As for packages getting updated unintentionally, easy_install and pip both require an argument to upgrade any existing packages (I think -U), so I am not sure how you are running into such a situation.
Quite easily, actually. I ran into pip wanting to upgrade numpy when I was installing/upgrading a package depending on numpy. Problem is, -U upgrades both the package you explicitly select *and* its dependencies. I know there's some way around this, but it's not obvious -- at least not for users. Cheers, Andreas.
On Fri, Sep 21, 2012 at 10:37 PM, Benjamin Root <ben.root@ou.edu> wrote:
On Fri, Sep 21, 2012 at 4:19 PM, Travis Oliphant <travis@continuum.io>wrote:
On Sep 21, 2012, at 3:13 PM, Ralf Gommers wrote:
Hi,
An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6']
in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that ( https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list.
The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason.
This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py:
try: import numpy except ImportError: raise ImportError("my_package requires numpy")
or
try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion)
Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere?
I agree. I would recommend against using install requires.
-Travis
Why? I have personally never had an issue with this. The only way I could imagine that this wouldn't work is if numpy was installed via some other means and there wasn't an entry in the easy-install.pth (or whatever equivalent pip uses).
Eh, just installing numpy with "python setup.py install" uses plain distutils, not setuptools. So there indeed isn't an entry in easy-install.pth. Which some consider a feature:)
If pip is having a problem detecting numpy, then that is a bug that needs fixing somewhere.
Sure. But who's going to do that?
As for packages getting updated unintentionally, easy_install and pip both require an argument to upgrade any existing packages (I think -U), so I am not sure how you are running into such a situation.
No, if the version detection fails pip will happily "upgrade" my 1.8.0-dev to 1.6.2.
I have found install_requires to be a powerful feature in my setup.py scripts, and I have seen no reason to discourage it. Perhaps I am the only one?
I'm sure you're not the only one. But it's still severely broken. Ralf
On Fri, Sep 21, 2012 at 9:42 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Eh, just installing numpy with "python setup.py install" uses plain distutils, not setuptools. So there indeed isn't an entry in easy-install.pth. Which some consider a feature:)
I don't think this is correct. To be clear on the technical issue: what's going on is that when pip sees install_requires=["numpy"], it needs to check whether you already have the distribution called "numpy" installed. It turns out that in the wonderful world of python packaging, "distributions" are not quite the same as "packages", so it can't do this by searching PYTHONPATH for a "numpy" directory. What it does is search PYTHONPATH for a file named numpy-<version-number->.egg-info[1]. This isn't *quite* as dumb as it seems, because in practice there really isn't a 1-to-1 mapping between source distributions and installed packages, but it's... pretty dumb. Anyway. The problem is that Ralf installed numpy by doing an in-place build in his source tree, and then adding his source tree to his PYTHONPATH. But, he didn't put a .egg-info on his PYTHONPATH, so pip couldn't tell that numpy was installed, and did something dumb. So the question is, how do we get a .egg-info? For the specific case Ralf ran into, I'm pretty sure the solution is just that if you're clever enough to do an in-place build and add it to your PYTHONPATH, you should be clever enough to also run 'python setupegg.py egg_info' which will create a .egg-info to go with your in-place build and everything will be fine. The question is whether there are any other situations where this can break. I'm not aware of any. Contrary to what's claimed in the bit I quoted above, I just ran a plain vanilla 'python setup.py install' on numpy inside a virtualenv, and I ended up with a .egg-info installed. I'm pretty sure plain old distutils installs .egg-infos these days too. In that bug report Ralf says there's some problem with virtualenvs, but I'm not sure what (I use virtualenvs extensively and have never run into anything). Can anyone elaborate? [1] or several other variants, see some PEP or another for the tedious details. -n P.S.: yeah the thing where pip decides to upgrade the world is REALLY OBNOXIOUS. It also appears to be on the list to be fixed in the next release or the next release+1, so I guess there's hope?: https://github.com/pypa/pip/pull/571
On Fri, Sep 21, 2012 at 5:39 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Sep 21, 2012 at 9:42 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Eh, just installing numpy with "python setup.py install" uses plain distutils, not setuptools. So there indeed isn't an entry in easy-install.pth. Which some consider a feature:)
I don't think this is correct. To be clear on the technical issue: what's going on is that when pip sees install_requires=["numpy"], it needs to check whether you already have the distribution called "numpy" installed. It turns out that in the wonderful world of python packaging, "distributions" are not quite the same as "packages", so it can't do this by searching PYTHONPATH for a "numpy" directory. What it does is search PYTHONPATH for a file named numpy-<version-number->.egg-info[1]. This isn't *quite* as dumb as it seems, because in practice there really isn't a 1-to-1 mapping between source distributions and installed packages, but it's... pretty dumb. Anyway. The problem is that Ralf installed numpy by doing an in-place build in his source tree, and then adding his source tree to his PYTHONPATH. But, he didn't put a .egg-info on his PYTHONPATH, so pip couldn't tell that numpy was installed, and did something dumb.
So the question is, how do we get a .egg-info? For the specific case Ralf ran into, I'm pretty sure the solution is just that if you're clever enough to do an in-place build and add it to your PYTHONPATH, you should be clever enough to also run 'python setupegg.py egg_info' which will create a .egg-info to go with your in-place build and everything will be fine.
The question is whether there are any other situations where this can break. I'm not aware of any. Contrary to what's claimed in the bit I quoted above, I just ran a plain vanilla 'python setup.py install' on numpy inside a virtualenv, and I ended up with a .egg-info installed. I'm pretty sure plain old distutils installs .egg-infos these days too. In that bug report Ralf says there's some problem with virtualenvs, but I'm not sure what (I use virtualenvs extensively and have never run into anything). Can anyone elaborate?
[1] or several other variants, see some PEP or another for the tedious details.
-n
P.S.: yeah the thing where pip decides to upgrade the world is REALLY OBNOXIOUS. It also appears to be on the list to be fixed in the next release or the next release+1, so I guess there's hope?: https://github.com/pypa/pip/pull/571
In statsmodels we moved to the check that Ralf proposes, and no requires. When I'm easy_installing a package I always need to watch out when a package tries to upgrade numpy. I just had to hit Crtl-C several times when the requires of pandas tried to update my numpy version. Josef
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Sep 21, 2012 at 11:39 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Sep 21, 2012 at 9:42 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Eh, just installing numpy with "python setup.py install" uses plain distutils, not setuptools. So there indeed isn't an entry in easy-install.pth. Which some consider a feature:)
I don't think this is correct. To be clear on the technical issue: what's going on is that when pip sees install_requires=["numpy"], it needs to check whether you already have the distribution called "numpy" installed. It turns out that in the wonderful world of python packaging, "distributions" are not quite the same as "packages", so it can't do this by searching PYTHONPATH for a "numpy" directory. What it does is search PYTHONPATH for a file named numpy-<version-number->.egg-info[1]. This isn't *quite* as dumb as it seems, because in practice there really isn't a 1-to-1 mapping between source distributions and installed packages, but it's... pretty dumb. Anyway. The problem is that Ralf installed numpy by doing an in-place build in his source tree, and then adding his source tree to his PYTHONPATH. But, he didn't put a .egg-info on his PYTHONPATH, so pip couldn't tell that numpy was installed, and did something dumb.
So the question is, how do we get a .egg-info? For the specific case Ralf ran into, I'm pretty sure the solution is just that if you're clever enough to do an in-place build and add it to your PYTHONPATH, you should be clever enough to also run 'python setupegg.py egg_info' which will create a .egg-info to go with your in-place build and everything will be fine.
That command first starts rebuilding numpy. The correct one seems to be 'python setupegg.py install_egg_info'. This does install the egg_info file in site-packages, but it's still not working: $ python -c "import numpy as np; print(np.__version__)" 1.8.0.dev-d8988ab $ ls /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ ... numpy-1.8.0.dev_d8988ab-py2.6.egg-info ... $ pip install -U --no-deps pandas Exception: Traceback (most recent call last): ... VersionConflict: (numpy 1.5.1 (/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages), Requirement.parse('numpy>=1.6')). As long as you try to do anything with PYTHONPATH, I think pip/easy_install/setuptools are broken in a quite fundamental way. The question is whether there are any other situations where this can
break. I'm not aware of any. Contrary to what's claimed in the bit I quoted above, I just ran a plain vanilla 'python setup.py install' on numpy inside a virtualenv, and I ended up with a .egg-info installed. I'm pretty sure plain old distutils installs .egg-infos these days too.
You're right, that is the case.
In that bug report Ralf says there's some problem with virtualenvs, but I'm not sure what (I use virtualenvs extensively and have never run into anything). Can anyone elaborate?
I haven't used them in a while, so I can't explain in detail now. Basic numpy install into virtualenvs is working now AFAIK (which was quite painful too), but I remember having problems when using them in combination with PYTHONPATH too.
[1] or several other variants, see some PEP or another for the tedious details.
-n
P.S.: yeah the thing where pip decides to upgrade the world is REALLY OBNOXIOUS. It also appears to be on the list to be fixed in the next release or the next release+1, so I guess there's hope?: https://github.com/pypa/pip/pull/571
Good to know. Let's hope that does make it in. Given it's development model, I'm less optimistic that easy_install will receive the same fix though .... Until both pip and easy_install are fixed, this alone should be enough for the advice to be "don't use install_requires". It's not like my alternative suggestion takes away any information or valuable functionality. Ralf
On Sat, Sep 22, 2012 at 1:18 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Sep 21, 2012 at 11:39 PM, Nathaniel Smith <njs@pobox.com> wrote:
So the question is, how do we get a .egg-info? For the specific case Ralf ran into, I'm pretty sure the solution is just that if you're clever enough to do an in-place build and add it to your PYTHONPATH, you should be clever enough to also run 'python setupegg.py egg_info' which will create a .egg-info to go with your in-place build and everything will be fine.
That command first starts rebuilding numpy.
No, it just seems to run the config and source-generation bits, not build anything. It also leaves the .egg-info in the source directory, which is what you want.
The correct one seems to be 'python setupegg.py install_egg_info'. This does install the egg_info file in site-packages, but it's still not working:
$ python -c "import numpy as np; print(np.__version__)" 1.8.0.dev-d8988ab $ ls /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ ... numpy-1.8.0.dev_d8988ab-py2.6.egg-info ... $ pip install -U --no-deps pandas Exception: Traceback (most recent call last): ... VersionConflict: (numpy 1.5.1 (/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages), Requirement.parse('numpy>=1.6')).
The problem here is that you have numpy 1.5.1 installed in a directory that appears on your PYTHONPATH *before* the directory that you installed the .egg-info into. The .egg-info is supposed to go in the same directory as the package; that way 'import numpy' and pip will always find corresponding versions of numpy, no matter how you change your PYTHONPATH. It does look like you can also use: python setup.py install_egg_info -d . This just uses standard distutils and skips running the config/source-generation step. (I guess this is because the vanilla distutils egg-info metadata is less thorough, and doesn't include a list of installed files.)
As long as you try to do anything with PYTHONPATH, I think pip/easy_install/setuptools are broken in a quite fundamental way.
I still see no evidence of this. Just put the .egg-info next to the package and move on...
P.S.: yeah the thing where pip decides to upgrade the world is REALLY OBNOXIOUS. It also appears to be on the list to be fixed in the next release or the next release+1, so I guess there's hope?: https://github.com/pypa/pip/pull/571
Good to know. Let's hope that does make it in. Given it's development model, I'm less optimistic that easy_install will receive the same fix though ....
Yeah, easy_install is abandoned and bit-rotting, which is why people usually recommend pip :-). But in this case, I thought that easy_install already doesn't upgrade the world when it runs? Is there something to fix here?
Until both pip and easy_install are fixed, this alone should be enough for the advice to be "don't use install_requires". It's not like my alternative suggestion takes away any information or valuable functionality.
pandas, for example, requires several other packages, and I found it quite convenient the other day when I wanted to try out a new version and pip automatically took care of setting all that up for me. It even correctly upgraded numpy, since the virtualenv I was using for testing had inherited my system-installed 1.5.2, but this was the first version of pandas that needed 1.6. Python packaging tools make me feel grumpy and traumatized too but I don't see how the solution is to just give up on computer-readable dependency-tracking altogether. -n
On Sun, Sep 23, 2012 at 10:20 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Sep 22, 2012 at 1:18 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Sep 21, 2012 at 11:39 PM, Nathaniel Smith <njs@pobox.com> wrote:
So the question is, how do we get a .egg-info? For the specific case Ralf ran into, I'm pretty sure the solution is just that if you're clever enough to do an in-place build and add it to your PYTHONPATH, you should be clever enough to also run 'python setupegg.py egg_info' which will create a .egg-info to go with your in-place build and everything will be fine.
That command first starts rebuilding numpy.
No, it just seems to run the config and source-generation bits, not build anything. It also leaves the .egg-info in the source directory, which is what you want.
You're right, sorry. I saw output like "building extension "numpy.core._dotblas" sources" scrolling by and hit Ctrl-C.
P.S.: yeah the thing where pip decides to upgrade the world is REALLY OBNOXIOUS. It also appears to be on the list to be fixed in the next release or the next release+1, so I guess there's hope?: https://github.com/pypa/pip/pull/571
Good to know. Let's hope that does make it in. Given it's development model, I'm less optimistic that easy_install will receive the same fix though ....
Yeah, easy_install is abandoned and bit-rotting, which is why people usually recommend pip :-). But in this case, I thought that easy_install already doesn't upgrade the world when it runs? Is there something to fix here?
It does, as Josef said above. It has the same -U and --no-deps flags.
Until both pip and easy_install are fixed, this alone should be enough for the advice to be "don't use install_requires". It's not like my alternative suggestion takes away any information or valuable functionality.
pandas, for example, requires several other packages, and I found it quite convenient the other day when I wanted to try out a new version and pip automatically took care of setting all that up for me. It even correctly upgraded numpy, since the virtualenv I was using for testing had inherited my system-installed 1.5.2, but this was the first version of pandas that needed 1.6.
So this saved you from reading "pandas requires numpy >= 1.6.1" and typing "pip install -U numpy". Not my definition of valuable functionality, and certainly not worth the risk of upgrading numpy silently for users. Python packaging tools make me feel grumpy and traumatized too but I
don't see how the solution is to just give up on computer-readable dependency-tracking altogether.
Proper dependency tracking would be preferable, but none at all is better than the current situation imho. Ralf
On Fri, Sep 21, 2012 at 4:37 PM, Benjamin Root <ben.root@ou.edu> wrote:
On Fri, Sep 21, 2012 at 4:19 PM, Travis Oliphant <travis@continuum.io> wrote:
On Sep 21, 2012, at 3:13 PM, Ralf Gommers wrote:
Hi,
An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6']
in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that (https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list.
The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason.
This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py:
try: import numpy except ImportError: raise ImportError("my_package requires numpy")
or
try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion)
Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere?
I agree. I would recommend against using install requires.
-Travis
Why? I have personally never had an issue with this. The only way I could imagine that this wouldn't work is if numpy was installed via some other means and there wasn't an entry in the easy-install.pth (or whatever equivalent pip uses). If pip is having a problem detecting numpy, then that is a bug that needs fixing somewhere.
As for packages getting updated unintentionally, easy_install and pip both require an argument to upgrade any existing packages (I think -U), so I am not sure how you are running into such a situation.
If a user use that option, it will also try to updaet NumPy. This is a bad default behavior. The work aroud is to pass -U and --no-deps to don't update the dependency. People don't want to update numpy when they update there package other package as Theano.
I have found install_requires to be a powerful feature in my setup.py scripts, and I have seen no reason to discourage it. Perhaps I am the only one?
What about if numpy is installed and recent enough, don't put in in the install_require. If not there, add it there? It will still fail if not c compiler is there, but maybe it won't update it at then same time? Fred
participants (7)
-
Andreas Hilboll -
Benjamin Root -
Frédéric Bastien -
josef.pktd＠gmail.com -
Nathaniel Smith -
Ralf Gommers -
Travis Oliphant