Robert Kern wrote:

Ian Bicking wrote:

...

I've created a script to set up a non-root installation of Python, suitable for testing installation procedures. It follows the instructions located here: http://peak.telecommunity.com/DevCenter/EasyInstall#non-root-installation

I really dislike this approach. I'm afraid that I don't understand what problem it's trying to solve. On OS X, the instructions tell you to install to ~/Library/Python/2.4/site-packages . Why do the general UNIX instructions tell you to make a ton of symlinks? IMHO, better advice would be (thanks in part to Prabhu Ramachandran):

Make a directory to hold your EasyInstall'ed packages (e.g. ~/lib/python2.4/site-packages). easy_install uses .pth tricks. Therefore, the directory where eggs are installed require that .pth files work there. On non-system wide installs, this can be achieved using a sitecustomize.py somewhere inside PYTHONPATH with the following lines:

import site, os site.addsitedir(os.path.expanduser('~/lib/python2.4/site-packages'))

easy_install needs to be told to use this directory and that it is okay to use. Add the following options to your ~/.pydistutils.cfg file:

[install] install_lib=~/lib/python$py_version_short/site-packages install_scripts=~/bin

[easy_install] site_dirs=~/lib/python$py_version_short/site-packages

This is what I am doing to get easy_install to install in /usr/local, and other private directories on the system. However, I find the non-root install useful for testing, as it is easy to recreate, and avoids side-effects from the rest of the system. Much of the time you actually want side effects, which is what you get with the other procedure. Anyway, I think it would be a good idea to have this in the setuptools documentation (it's a wiki, so if you want to add it I think you should feel free). My own description of the process is at: http://blog.ianbicking.org/alternate-python-install-dir.html One issue with a number of these options, is that I don't think easy_install expands ~ or $py_version_short in all its variables. I think that can just be classified a bug, but in practice it will cause problems right now.

At this point in time, I think this approach is cleaner. Eventually it might run into conflicts with a distribution-provided easy_install, but AFAICT there aren't any of those yet.

In any case, neither of these approaches is going to be tenable in the long term. Having the non-root user recreate the whole python distribution via symlinks isn't a reasonable approach. Controlling the installation/activation of eggs via .pth files was a cool approach, but it appears that it's not terribly flexible because of the way that Python inserts them into sys.path.

A more general way forward might be to have sitecustomize.py (or site.py if your distro is kind enough to take care of such things for you) read data files that list the eggs that are activated.

In practice, easy_install.pth is a list of eggs that are activated. Though it's a little fuzzy, because as eggs they aren't yet activated, even though as Python packages they are available. And I think some of the issue PJE was having with how scripts should install, is what do you do when the egg on sys.path isn't the egg you want to activate. (Can that entry simply be removed from sys.path, and sys.modules checked to make sure nothing from that egg has been imported yet?)

sitecustomize.py could then insert the eggs wherever it sees fit into sys.path. On Debian, for example, there would be a file in /usr/lib/pythonX.Y/site-packages/ for eggs that are provided by real dpkg-installed .deb's. There would be another file in /usr/local/lib/pythonX.Y/site-packages/ for eggs that are installed by root but aren't real Debian packages.

A .pth file can point to paths anywhere, so a single .pth file (even in, say, /etc/python/packages.pth) could point anywhere in the system, including /usr/lib and /usr/local/lib. It couldn't point to per-user installs, though...

There might be a final one somewhere canonical in a user's home directory (~/.easy_install perhaps) for eggs that are local to the user (and probably installed to ~/lib/pythonX.Y/site-packages/ but not necessarily). These data files would be searched in reverse order: ~/, /usr/local/, /usr/. It would be nice if the data files could specify that certain eggs in the later directories are to be ignored.

It would be nice if site(customize).py added ~/lib/pythonX.Y to the path as well, and any .pth files there as well. It would also be nice if the list of directories where .pth files were read was kept somewhere, like say site.sitedirs. These are fairly low-impact changes, I think. In some ways, I would go so far as to say that distribution maintainers have simply ignored these issues, rather than made any explicit (correct or incorrect) choice. For instance, Debian should have a distutils.cfg that installs packages into /usr/local/lib/pythonX.Y. Maybe I should submit that as a bug, because without it distutils simply doesn't meet Debian policy. So, while these are changes to system layout, I think there's little reason for the status quo in many cases.

Does this sound like a workable idea?

On a similar note: easy_install does a lot of sanity checks to determine if it ought to install to the given install_lib directory. Unfortunately, those sanity checks can be too restrictive. For example, when I use Debian, I manage my /usr/local/ tree with GNU Stow. I want to put everything managed by easy_install under the prefix /usr/local/stow/easy_install/ (with subdirectories bin/, lib/python2.4/site-packages/, and so on). When I run Stow, everything will be symlinked into /usr/local/. Because Debian was thoughtful, /usr/local/lib/python2.4/site-packages/ is already on the PYTHONPATH and is .pth-enabled so everything would work just fine. Unfortunately, /usr/local/stow/easy_install/lib/python2.4/site-packages/ isn't on the PYTHONPATH and easy_install will refuse to install there even when it has been specified as a valid --site-dir.

There's also an issue where easy_install has a site_dir option, but no other installation methods do (e.g., setup.py install, setup.py develop).

Paranoia is healthy, but we need a way to override the paranoia. I am requesting a --force option to override this sanity check. Preferably there would be one --force option per sanity check and one overall --force that entails all of the others, but I'll take what I can get. I'm willing to work on a patch if given a little guidance as to where all of the sanity checks are.

I'd like the whole thing to be more expansive. That is, not to have it as an easy_install option, but as an option that applies to all of distutils (or at least all of setuptools). In this case, both a global option for site_dirs, and a global option for the sys.path (in this case you say it should treat your stow directory as being on sys.path).