[IPython-dev] Weird PYTHONPATH, etc. issue

Erik Bray erik.m.bray at gmail.com
Thu Sep 27 08:59:05 EDT 2012


On Tue, Sep 25, 2012 at 9:50 PM, David Warde-Farley
<d.warde.farley at gmail.com> wrote:
> On Tue, Sep 25, 2012 at 2:18 PM, Fernando Perez <fperez.net at gmail.com> wrote:
>> On Tue, Sep 25, 2012 at 5:41 AM, Thomas Spura <tomspur at fedoraproject.org> wrote:
>>> I never had a problem with this setup so far, unless you count the
>>> circumvention of the sysadmin a problem...
>>> I think it's easier to install anything I need in userspace _again_
>>> and you can be sure, that it's consistent and working, but a small
>>> overhead is added of course.
>>
>> But is it possible with a virtualenv to benefit from system-wide
>> packages while overriding a few yourself, if the system-wide ones use
>> the underhanded setutpools behavior?
>>
>> Because that's the problem that I've found to be hard to manage with
>> setuptools: wanting to keep the system-wide stuff but overlay a few
>> personally managed packages to override system ones.  When the system
>> ones use regular distutils, that's trivial to do.  But with setuptools
>> messing with sys.path at runtime, you end up with the conundrum David
>> is in...
>>
>> Maybe virtualenvs are the solution, it's still unfortunate an entire
>> new system has to be devised to protect against setuptools
>> aggressively malicious behavior.
>
> Indeed. It turns out, our sysadmin had run into this problem before
> with other users but didn't know how to solve it. I told him to
> replace easy_install with pip in his site-packages generation script,
> and he did so with his install of ipython (planning to fix other
> packages when he updates them or when people wish to override them).
> Problem solved. Down with setuptools!

I don't really see this so much as a setuptools problem though.  Your
sysadmin shouldn't be using PYTHONPATH for this.  I've had to battle
this a bit at my own institution.  The problem with using PYTHONPATH
to add site-specific site-packages directories are many-fold.  For
one, if the user then messes with their PYTHONPATH they suddenly can't
`import numpy` anymore and come complaining that numpy broke or
something like that.  PYTHONPATH should be there as an easy way for
the user to optionally add additional paths to the front of their
sys.path.  So when it's used like this it basically takes away your
freedom to customize anything on your path at all since the contents
of PYTHONPATH come first.  Sure you can always add your stuff to the
front of PYTHONPATH too, but a lot of users get confused by that.  Or
it doesn't work with virtualenv or buildout, or other such problems.

What the sysadmin *should* be doing is adding a sitecustomize.py
somewhere on the standard sys.path (in lib/pythonX.Y/site-packages for
example) and adding any custom paths there, while leaving your
PYTHONPATH alone.  As mentioned elsewhere in this thread
usercustomize.py can also be used.

Don't get me wrong--setuptools can cause its share of problems. But I
really don't see this as a setuptools fault so much as a site
configuration mess.

Erik



More information about the IPython-dev mailing list