[Distutils] Working environment
ianb at colorstudy.com
Tue Mar 7 19:01:55 CET 2006
So, coming back to the idea of a working environment, an isolated and
more-or-less self-contained environment for holding installed packages.
Sorry if this is a little scattered. I'm just summarizing my thoughts
and the open issues I see, in no particular order (because I'm not sure
what order to approach these things in).
I'm assuming such an environment will be encapsulated in a single
directory, looking something like:
The conf/ directory doesn't really relate to much of this specifically,
but in many situations it would be useful. Depending on the situation,
other subdirectories may exist.
Each of the scripts in bin/ should know what their working environment
is. This is slightly tricky, depending on what that means. If it is a
totally isolated environment -- no site-packages on sys.path -- then I
feel like the script wrappers have to be shell scripts, to invoke Python
with -S (which is hard to do portably on the #! line). I don't know the
details of doing the same thing on Windows, but I assume it is possible.
The actual directory location should be portable -- all paths should
be relative, and you should be able to move the directory around.
lib/python2.4/ is for packages. I'm almost inclined to say that
--single-version-externally-managed makes sense on some level, with a
record kept in some standard place (lib/python2.4/install-record.txt?)
-- but I'm basically indifferent. I at least don't see a need for
multiple simultaneous versions in this setup, and multiple versions do
lead to confusion. Version metadata is still nice, of course.
src/ is for checkouts, where each package is installed with setup.py
develop. These are naturally single-version, which is part of why I
like the idea of only using single-version setups. I'm a little unsure
of how src/ should be layed out. In practice I want all "my" packages
to be installed in src/ as checkouts, either from tags or the trunk (or
a branch or whatever). So I'm not sure if I should name the
subdirectories after the package, or maybe even the package plus a tag
One of the things SwitchTower (now "Cappucino", I think) does in Rails
land is it makes a dated checkout, then activates that checkout (it does
it with a symlink, we'd do it with setup.py develop). It then rolls
back by switching to an existing checkout. Of course svn switch + svn
up does this in place, and with less checkout trash laying around, even
if rollbacks aren't as fast as a result. So, I'm thinking just
There's an installation issue there -- it would be nice if I could say
"these are the packages I want to install as editable" and easy_install
would pick those up (maybe detecting based on what package index the
package was found in) and install them in src/ as editable.
sys.path would contain /usr/lib/python2.4, optionally
/usr/lib/python2.4/site-packages, and env/lib/python2.4/, and all the
similar directories. Unfortunately figuring out what "similar"
directories there are is hard. sys.path on my machine now has 63
entries normally and 12 with python -S. I guess I'd really like to
start with 12 and build up, instead of 63 and try to strip them down.
Installation as a whole is an open issue. Putting in env/setup.cfg with
the setting specific to that working environment works to a degree --
easy_install will pick it up if invoked from there. But that doesn't
work with setup.py develop, or setup.py install, or some other
scenarios. The system distutils.cfg doesn't really work, because the
only expansion it knows how to do is of user directories, so there's
little way to pass interesting information in (like a "this is my
setup.cfg" environmental variable or something). Maybe with PYTHONPATH
to indicate the working environment, and a distutils monkeypatch put
into lib/python2.4/distutils/__init__.py? I played around with putting
the path setup in sitecustomize, but that runs after site.py, and
doesn't run at all if python -S is used, so it seems like it brings in
too much before it can remove stuff.
Another option is a completely new python interpreter bound to the
environment. Basically the virtual-python.py option
In this model using env/bin/python indicate the proper environment,
and you'd have local installs of *everything* including easy_install.
This fixes so many problems without crazy hacks that it strongly appeals
to me, especially if we can make it somewhat lighter. I get this
/usr/lib/python2.4.zip on my path, that doesn't usually exist (does it
ever get created by default); if we could create that somehow on demand
and use such a big-bundle-zip, that seems lighter and faster and nicer.
If we just put .pyc files in it, and those .pyc files refer back to
the actual module source (in /usr/lib/python2.4/), then tracebacks
should also still work, right? No actual symlinks either, so it should
work on Windows. I'm not entirely sure where I'm going with this, though.
Sorry for the length. I've been stewing on this without a lot of
progress since PyCon so I thought I'd just throw out my current
thoughts. Maybe what I really want to do is hack on virtual-python.py
some more and see where that gets me.
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
More information about the Distutils-SIG