[Distutils] DWIM installation with setuptools

[Phillip J. Eby]

Okay, so I've been thinking about all the recent complaints/requests 
regarding the perils of PYTHONPATH, site-packages, --prefix, and all 
that.  And I'm thinking, is there some way I can work around all this stuff 
so that installation is totally DWIMmish -- that is, that the install can 
just "do what I mean", while still allowing a default version of a package 
to be set.

This already works perfectly with root access or a virtual Python, so we 
can pretty much ignore those setups for this discussion (except to make 
sure we don't break them).

So, mostly what's left to worry about is being able to select an arbitrary 
installation location, and still have it work.  If we assume you're smart 
enough to pick a location that will already be on your PYTHONPATH (and we 
could verify that by checking that it already is), then all we have to 
worry about is making sure that the right version of a package gets added 
to sys.path at runtime.

For scripts this isn't that big of a deal.  I could create a specialized 
bootstrap loader version of pkg_resources.py and dump it alongside any 
scripts installed by easy_install.  It'd be a pain to implement, but I 
could probably make it work such that sys.path was searched for a 
setuptools egg, and then have that get used to bootstrap the rest.

What this doesn't work for is the interactive interpreter, or code that 
doesn't use require().  Even if the relevant directory is on sys.path, 
Python doesn't normally read .pth files in PYTHONPATH directories.  I've 
also learned that many Linux distros appear to hack sys.path in odd ways, 
and I'm not positive that all of them process .pth files in the directories 
they add.  Certainly they don't support .pth for PYTHONPATH directories.

Since it's not just the interactive interpreter, hacking .pythonrc isn't 
going to help much.  That leaves the 'site' and 'sitecustomize' 
modules.  The current PYTHONPATH install support hijacks the 'site' module 
with a special .pth-honoring feature, so as long as you have the setuptools 
egg listed directly in PYTHONPATH, you're golden.

But alas, that isn't DWIM, since you now have to manipulate the PYTHONPATH 
*before* you can reasonably do anything.  Part of the essence of what we 
desire is that any required steps be done for you *during* the installation 
rather than requiring an extra step before or afterwards.

Unfortunately, this means that hacking 'site' can't work unless we actually 
install a site.py.  Will that work?

Well, yes and no.  It appears Python will recognize an added site.py that 
appears in a PYTHONPATH, but many platforms will not recognize a site.py 
that's in the script directory but not PYTHONPATH.  Windows seems to be the 
only platform where putting a site.py in the script directory has any effect.

This is really good news, however.  It appears to mean that slapping a copy 
of my hacked 'site.py' (i.e., the one that honors .pth files along 
PYTHONPATH) into the easy_install target directory would make 
PYTHONPATH-based installs essentially DWIMmish.  To complete the illusion, 
I'd need to add some code to script wrappers to process the setuptools.pth 
file in the script directory if pkg_resources can't be otherwise imported.

We could then lobby for the Python 2.5 site.py to behave the same way as my 
hacked one does: i.e., process .pth files found in PYTHONPATH directories, 
inserting them ahead of site-packages and its ilk, so that it wouldn't be 
necessary to install the hacked site.py in such cases.  (It's also not 
necessary to install it in known "site" directories, such as site-packages 
and any directories listed in --site-dirs.  But even if it's installed 
unnecessarily, it shouldn't hurt anything.)

Since 'site' is a stdlib module and isn't normally intended for 
customization, it shouldn't conflict with any user-installed 
modules.  However, easy_install could detect an existing site.py and warn 
if its content is different from the one easy_install wants to use.

I'm trying to remember why I didn't do this around the time I implemented 
the original PYTHONPATH support (which first required a hacked 'site' 
module).  I think I was thrown by the lack of cross-platform script 
directory support for importing 'site', and it didn't occur to me that I 
could easily work around that from within scripts.  Also, it's easy to 
mentally confuse the script directory and the install directory while 
thinking about some of this stuff.

But as long as the install directory (irrespective of the script directory) 
is on PYTHONPATH and has a hacked site.py, what's in the script directory 
doesn't matter.  The only case where the script directory matters is if:

1. it's the same directory the libraries were installed to,
2. it's not on PYTHONPATH, and
3. the user puts a script of their own in there and expects it to work 
anywhere but Windows

This edge case could be handled, however, by refusing to install libraries 
to non-PYTHONPATH directories unless you're on Windows or you use 
--multi-version.  I'm somewhat uncertain about this, though, because it 
doesn't seem to let you create a self-sufficient application directory.

Right now, you can use "easy_install -ad somedir ..." to copy all needed 
eggs and scripts to the target directory, making a self-contained 
application directory that has everything it needs to run.  OTOH, I suppose 
if you just made that -mad instead of -ad, it would work just fine for that 
purpose under the proposed new regime.

So, to sum up, here are the changes that would take place:

* easy_install would treat all PYTHONPATH-specified directories as though 
they were listed in --site-dirs, but it would also copy and compile a 
hacked site.py into them during installation, if they aren't *actually* 
listed in --site-dirs or in setuptools' default computation of "site" 
directories.

* Currently, easy_install defaults to -m if you install to a non-"site" 
directory; it would now instead stop and refuse to install to such 
directories unless you explicitly specify -m.

* easy_install could now meaningfully grow a --prefix option, since it will 
now DWIM as long as the target dir is recognizable as (or convertible to) a 
"site" directory.  And, in the cases where DWIM can't be guaranteed, it 
will stop and inform the user that they need to fix PYTHONPATH, 
--site-dirs, or use -m/--multi-version mode.

What do y'all think?  Do we have a winner yet?