On Fri, Sep 21, 2012 at 11:39 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Sep 21, 2012 at 9:42 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
> Eh, just installing numpy with "python setup.py install" uses plain
> distutils, not setuptools. So there indeed isn't an entry in
> easy-install.pth.  Which some consider a feature:)

I don't think this is correct. To be clear on the technical issue:
what's going on is that when pip sees install_requires=["numpy"], it
needs to check whether you already have the distribution called
"numpy" installed. It turns out that in the wonderful world of python
packaging, "distributions" are not quite the same as "packages", so it
can't do this by searching PYTHONPATH for a "numpy" directory. What it
does is search PYTHONPATH for a file named
numpy-<version-number->.egg-info[1]. This isn't *quite* as dumb as it
seems, because in practice there really isn't a 1-to-1 mapping between
source distributions and installed packages, but it's... pretty dumb.
Anyway. The problem is that Ralf installed numpy by doing an in-place
build in his source tree, and then adding his source tree to his
PYTHONPATH. But, he didn't put a .egg-info on his PYTHONPATH, so pip
couldn't tell that numpy was installed, and did something dumb.

So the question is, how do we get a .egg-info? For the specific case
Ralf ran into, I'm pretty sure the solution is just that if you're
clever enough to do an in-place build and add it to your PYTHONPATH,
you should be clever enough to also run 'python setupegg.py egg_info'
which will create a .egg-info to go with your in-place build and
everything will be fine.

That command first starts rebuilding numpy. The correct one seems to be 'python setupegg.py install_egg_info'. This does install the egg_info file in site-packages, but it's still not working:

$ python -c "import numpy as np; print(np.__version__)"
$ ls /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
$ pip install -U --no-deps pandas
Traceback (most recent call last):
VersionConflict: (numpy 1.5.1 (/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages), Requirement.parse('numpy>=1.6')).

As long as you try to do anything with PYTHONPATH, I think pip/easy_install/setuptools are broken in a quite fundamental way.

The question is whether there are any other situations where this can
break. I'm not aware of any. Contrary to what's claimed in the bit I
quoted above, I just ran a plain vanilla 'python setup.py install' on
numpy inside a virtualenv, and I ended up with a .egg-info installed.
I'm pretty sure plain old distutils installs .egg-infos these days

You're right, that is the case.
In that bug report Ralf says there's some problem with
virtualenvs, but I'm not sure what (I use virtualenvs extensively and
have never run into anything). Can anyone elaborate?

I haven't used them in a while, so I can't explain in detail now. Basic numpy install into virtualenvs is working now AFAIK (which was quite painful too), but I remember having problems when using them in combination with PYTHONPATH too.

[1] or several other variants, see some PEP or another for the tedious details.


P.S.: yeah the thing where pip decides to upgrade the world is REALLY
OBNOXIOUS. It also appears to be on the list to be fixed in the next
release or the next release+1, so I guess there's hope?:

Good to know. Let's hope that does make it in. Given it's development model, I'm less optimistic that easy_install will receive the same fix though ....  Until both pip and easy_install are fixed, this alone should be enough for the advice to be "don't use install_requires". It's not like my alternative suggestion takes away any information or valuable functionality.