[Distutils] Non-root install for testing

[Ian Bicking]

Robert Kern wrote:
> Ian Bicking wrote:
> 
>>I've created a script to set up a non-root installation of Python,
>>suitable for testing installation procedures.  It follows the
>>instructions located here:
>>http://peak.telecommunity.com/DevCenter/EasyInstall#non-root-installation
> 
> 
> I really dislike this approach. I'm afraid that I don't understand what
> problem it's trying to solve. On OS X, the instructions tell you to
> install to ~/Library/Python/2.4/site-packages . Why do the general UNIX
> instructions tell you to make a ton of symlinks? IMHO, better advice
> would be (thanks in part to Prabhu Ramachandran):
> 
>   Make a directory to hold your EasyInstall'ed packages (e.g.
>   ~/lib/python2.4/site-packages). easy_install uses .pth tricks.
>   Therefore, the directory where eggs are installed require that .pth
>   files work there.  On non-system wide installs, this can be achieved
>   using a sitecustomize.py somewhere inside PYTHONPATH with the
>   following lines:
> 
>     import site, os
>     site.addsitedir(os.path.expanduser('~/lib/python2.4/site-packages'))
> 
>   easy_install needs to be told to use this directory and that it is
>   okay to use. Add the following options to your ~/.pydistutils.cfg
>   file:
> 
>     [install]
>     install_lib=~/lib/python$py_version_short/site-packages
>     install_scripts=~/bin
> 
>     [easy_install]
>     site_dirs=~/lib/python$py_version_short/site-packages

This is what I am doing to get easy_install to install in /usr/local,
and other private directories on the system.  However, I find the
non-root install useful for testing, as it is easy to recreate, and
avoids side-effects from the rest of the system.  Much of the time you
actually want side effects, which is what you get with the other procedure.

Anyway, I think it would be a good idea to have this in the setuptools
documentation (it's a wiki, so if you want to add it I think you should
feel free).  My own description of the process is at:
http://blog.ianbicking.org/alternate-python-install-dir.html

One issue with a number of these options, is that I don't think
easy_install expands ~ or $py_version_short in all its variables.  I
think that can just be classified a bug, but in practice it will cause
problems right now.

> At this point in time, I think this approach is cleaner. Eventually it
> might run into conflicts with a distribution-provided easy_install, but
> AFAICT there aren't any of those yet.
> 
> In any case, neither of these approaches is going to be tenable in the
> long term. Having the non-root user recreate the whole python
> distribution via symlinks isn't a reasonable approach. Controlling the
> installation/activation of eggs via .pth files was a cool approach, but
> it appears that it's not terribly flexible because of the way that
> Python inserts them into sys.path.
> 
> A more general way forward might be to have sitecustomize.py (or site.py
> if your distro is kind enough to take care of such things for you) read
> data files that list the eggs that are activated. 

In practice, easy_install.pth is a list of eggs that are activated.
Though it's a little fuzzy, because as eggs they aren't yet activated,
even though as Python packages they are available.  And I think some of
the issue PJE was having with how scripts should install, is what do you
do when the egg on sys.path isn't the egg you want to activate.  (Can
that entry simply be removed from sys.path, and sys.modules checked to
make sure nothing from that egg has been imported yet?)

> sitecustomize.py could
> then insert the eggs wherever it sees fit into sys.path. On Debian, for
> example, there would be a file in /usr/lib/pythonX.Y/site-packages/ for
> eggs that are provided by real dpkg-installed .deb's. There would be
> another file in /usr/local/lib/pythonX.Y/site-packages/ for eggs that
> are installed by root but aren't real Debian packages. 

A .pth file can point to paths anywhere, so a single .pth file (even in,
say, /etc/python/packages.pth) could point anywhere in the system,
including /usr/lib and /usr/local/lib.  It couldn't point to per-user
installs, though...

> There might be a
> final one somewhere canonical in a user's home directory
> (~/.easy_install perhaps) for eggs that are local to the user (and
> probably installed to ~/lib/pythonX.Y/site-packages/ but not
> necessarily). These data files would be searched in reverse order: ~/,
> /usr/local/, /usr/. It would be nice if the data files could specify
> that certain eggs in the later directories are to be ignored.

It would be nice if site(customize).py added ~/lib/pythonX.Y to the path
as well, and any .pth files there as well.  It would also be nice if the
list of directories where .pth files were read was kept somewhere, like
say site.sitedirs.  These are fairly low-impact changes, I think.

In some ways, I would go so far as to say that distribution maintainers
have simply ignored these issues, rather than made any explicit (correct
or incorrect) choice.  For instance, Debian should have a distutils.cfg
that installs packages into /usr/local/lib/pythonX.Y.  Maybe I should
submit that as a bug, because without it distutils simply doesn't meet
Debian policy.  So, while these are changes to system layout, I think
there's little reason for the status quo in many cases.

> Does this sound like a workable idea?
> 
> On a similar note: easy_install does a lot of sanity checks to determine
> if it ought to install to the given install_lib directory.
> Unfortunately, those sanity checks can be too restrictive. For example,
> when I use Debian, I manage my /usr/local/ tree with GNU Stow. I want to
> put everything managed by easy_install under the prefix
> /usr/local/stow/easy_install/ (with subdirectories bin/,
> lib/python2.4/site-packages/, and so on). When I run Stow, everything
> will be symlinked into /usr/local/. Because Debian was thoughtful,
> /usr/local/lib/python2.4/site-packages/ is already on the PYTHONPATH and
> is .pth-enabled so everything would work just fine. Unfortunately,
> /usr/local/stow/easy_install/lib/python2.4/site-packages/ isn't on the
> PYTHONPATH and easy_install will refuse to install there even when it
> has been specified as a valid --site-dir.

There's also an issue where easy_install has a site_dir option, but no
other installation methods do (e.g., setup.py install, setup.py develop).

> Paranoia is healthy, but we need a way to override the paranoia. I am
> requesting a --force option to override this sanity check. Preferably
> there would be one --force option per sanity check and one overall
> --force that entails all of the others, but I'll take what I can get.
> I'm willing to work on a patch if given a little guidance as to where
> all of the sanity checks are.

I'd like the whole thing to be more expansive.  That is, not to have it
as an easy_install option, but as an option that applies to all of
distutils (or at least all of setuptools).  In this case, both a global
option for site_dirs, and a global option for the sys.path (in this case
you say it should treat your stow directory as being on sys.path).