[Distutils] DWIM installation with setuptools
Phillip J. Eby
pje at telecommunity.com
Thu Feb 9 23:03:57 CET 2006
Okay, so I've been thinking about all the recent complaints/requests
regarding the perils of PYTHONPATH, site-packages, --prefix, and all
that. And I'm thinking, is there some way I can work around all this stuff
so that installation is totally DWIMmish -- that is, that the install can
just "do what I mean", while still allowing a default version of a package
to be set.
This already works perfectly with root access or a virtual Python, so we
can pretty much ignore those setups for this discussion (except to make
sure we don't break them).
So, mostly what's left to worry about is being able to select an arbitrary
installation location, and still have it work. If we assume you're smart
enough to pick a location that will already be on your PYTHONPATH (and we
could verify that by checking that it already is), then all we have to
worry about is making sure that the right version of a package gets added
to sys.path at runtime.
For scripts this isn't that big of a deal. I could create a specialized
bootstrap loader version of pkg_resources.py and dump it alongside any
scripts installed by easy_install. It'd be a pain to implement, but I
could probably make it work such that sys.path was searched for a
setuptools egg, and then have that get used to bootstrap the rest.
What this doesn't work for is the interactive interpreter, or code that
doesn't use require(). Even if the relevant directory is on sys.path,
Python doesn't normally read .pth files in PYTHONPATH directories. I've
also learned that many Linux distros appear to hack sys.path in odd ways,
and I'm not positive that all of them process .pth files in the directories
they add. Certainly they don't support .pth for PYTHONPATH directories.
Since it's not just the interactive interpreter, hacking .pythonrc isn't
going to help much. That leaves the 'site' and 'sitecustomize'
modules. The current PYTHONPATH install support hijacks the 'site' module
with a special .pth-honoring feature, so as long as you have the setuptools
egg listed directly in PYTHONPATH, you're golden.
But alas, that isn't DWIM, since you now have to manipulate the PYTHONPATH
*before* you can reasonably do anything. Part of the essence of what we
desire is that any required steps be done for you *during* the installation
rather than requiring an extra step before or afterwards.
Unfortunately, this means that hacking 'site' can't work unless we actually
install a site.py. Will that work?
Well, yes and no. It appears Python will recognize an added site.py that
appears in a PYTHONPATH, but many platforms will not recognize a site.py
that's in the script directory but not PYTHONPATH. Windows seems to be the
only platform where putting a site.py in the script directory has any effect.
This is really good news, however. It appears to mean that slapping a copy
of my hacked 'site.py' (i.e., the one that honors .pth files along
PYTHONPATH) into the easy_install target directory would make
PYTHONPATH-based installs essentially DWIMmish. To complete the illusion,
I'd need to add some code to script wrappers to process the setuptools.pth
file in the script directory if pkg_resources can't be otherwise imported.
We could then lobby for the Python 2.5 site.py to behave the same way as my
hacked one does: i.e., process .pth files found in PYTHONPATH directories,
inserting them ahead of site-packages and its ilk, so that it wouldn't be
necessary to install the hacked site.py in such cases. (It's also not
necessary to install it in known "site" directories, such as site-packages
and any directories listed in --site-dirs. But even if it's installed
unnecessarily, it shouldn't hurt anything.)
Since 'site' is a stdlib module and isn't normally intended for
customization, it shouldn't conflict with any user-installed
modules. However, easy_install could detect an existing site.py and warn
if its content is different from the one easy_install wants to use.
I'm trying to remember why I didn't do this around the time I implemented
the original PYTHONPATH support (which first required a hacked 'site'
module). I think I was thrown by the lack of cross-platform script
directory support for importing 'site', and it didn't occur to me that I
could easily work around that from within scripts. Also, it's easy to
mentally confuse the script directory and the install directory while
thinking about some of this stuff.
But as long as the install directory (irrespective of the script directory)
is on PYTHONPATH and has a hacked site.py, what's in the script directory
doesn't matter. The only case where the script directory matters is if:
1. it's the same directory the libraries were installed to,
2. it's not on PYTHONPATH, and
3. the user puts a script of their own in there and expects it to work
anywhere but Windows
This edge case could be handled, however, by refusing to install libraries
to non-PYTHONPATH directories unless you're on Windows or you use
--multi-version. I'm somewhat uncertain about this, though, because it
doesn't seem to let you create a self-sufficient application directory.
Right now, you can use "easy_install -ad somedir ..." to copy all needed
eggs and scripts to the target directory, making a self-contained
application directory that has everything it needs to run. OTOH, I suppose
if you just made that -mad instead of -ad, it would work just fine for that
purpose under the proposed new regime.
So, to sum up, here are the changes that would take place:
* easy_install would treat all PYTHONPATH-specified directories as though
they were listed in --site-dirs, but it would also copy and compile a
hacked site.py into them during installation, if they aren't *actually*
listed in --site-dirs or in setuptools' default computation of "site"
* Currently, easy_install defaults to -m if you install to a non-"site"
directory; it would now instead stop and refuse to install to such
directories unless you explicitly specify -m.
* easy_install could now meaningfully grow a --prefix option, since it will
now DWIM as long as the target dir is recognizable as (or convertible to) a
"site" directory. And, in the cases where DWIM can't be guaranteed, it
will stop and inform the user that they need to fix PYTHONPATH,
--site-dirs, or use -m/--multi-version mode.
What do y'all think? Do we have a winner yet?
More information about the Distutils-SIG