On Fri, Sep 21, 2012 at 11:39 PM, Nathaniel Smith
On Fri, Sep 21, 2012 at 9:42 PM, Ralf Gommers
wrote: Eh, just installing numpy with "python setup.py install" uses plain distutils, not setuptools. So there indeed isn't an entry in easy-install.pth. Which some consider a feature:)
I don't think this is correct. To be clear on the technical issue: what's going on is that when pip sees install_requires=["numpy"], it needs to check whether you already have the distribution called "numpy" installed. It turns out that in the wonderful world of python packaging, "distributions" are not quite the same as "packages", so it can't do this by searching PYTHONPATH for a "numpy" directory. What it does is search PYTHONPATH for a file named numpy-<version-number->.egg-info[1]. This isn't *quite* as dumb as it seems, because in practice there really isn't a 1-to-1 mapping between source distributions and installed packages, but it's... pretty dumb. Anyway. The problem is that Ralf installed numpy by doing an in-place build in his source tree, and then adding his source tree to his PYTHONPATH. But, he didn't put a .egg-info on his PYTHONPATH, so pip couldn't tell that numpy was installed, and did something dumb.
So the question is, how do we get a .egg-info? For the specific case Ralf ran into, I'm pretty sure the solution is just that if you're clever enough to do an in-place build and add it to your PYTHONPATH, you should be clever enough to also run 'python setupegg.py egg_info' which will create a .egg-info to go with your in-place build and everything will be fine.
That command first starts rebuilding numpy. The correct one seems to be 'python setupegg.py install_egg_info'. This does install the egg_info file in site-packages, but it's still not working: $ python -c "import numpy as np; print(np.__version__)" 1.8.0.dev-d8988ab $ ls /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ ... numpy-1.8.0.dev_d8988ab-py2.6.egg-info ... $ pip install -U --no-deps pandas Exception: Traceback (most recent call last): ... VersionConflict: (numpy 1.5.1 (/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages), Requirement.parse('numpy>=1.6')). As long as you try to do anything with PYTHONPATH, I think pip/easy_install/setuptools are broken in a quite fundamental way. The question is whether there are any other situations where this can
break. I'm not aware of any. Contrary to what's claimed in the bit I quoted above, I just ran a plain vanilla 'python setup.py install' on numpy inside a virtualenv, and I ended up with a .egg-info installed. I'm pretty sure plain old distutils installs .egg-infos these days too.
You're right, that is the case.
In that bug report Ralf says there's some problem with virtualenvs, but I'm not sure what (I use virtualenvs extensively and have never run into anything). Can anyone elaborate?
I haven't used them in a while, so I can't explain in detail now. Basic numpy install into virtualenvs is working now AFAIK (which was quite painful too), but I remember having problems when using them in combination with PYTHONPATH too.
[1] or several other variants, see some PEP or another for the tedious details.
-n
P.S.: yeah the thing where pip decides to upgrade the world is REALLY OBNOXIOUS. It also appears to be on the list to be fixed in the next release or the next release+1, so I guess there's hope?: https://github.com/pypa/pip/pull/571
Good to know. Let's hope that does make it in. Given it's development model, I'm less optimistic that easy_install will receive the same fix though .... Until both pip and easy_install are fixed, this alone should be enough for the advice to be "don't use install_requires". It's not like my alternative suggestion takes away any information or valuable functionality. Ralf