[Distutils] Working environment

[Ian Bicking]

So, coming back to the idea of a working environment, an isolated and 
more-or-less self-contained environment for holding installed packages. 
  Sorry if this is a little scattered.  I'm just summarizing my thoughts 
and the open issues I see, in no particular order (because I'm not sure 
what order to approach these things in).

I'm assuming such an environment will be encapsulated in a single 
directory, looking something like:

env/
   bin/
   lib/python2.4/
   src/
   conf/

The conf/ directory doesn't really relate to much of this specifically, 
but in many situations it would be useful.  Depending on the situation, 
other subdirectories may exist.

Each of the scripts in bin/ should know what their working environment 
is.  This is slightly tricky, depending on what that means.  If it is a 
totally isolated environment -- no site-packages on sys.path -- then I 
feel like the script wrappers have to be shell scripts, to invoke Python 
with -S (which is hard to do portably on the #! line).  I don't know the 
details of doing the same thing on Windows, but I assume it is possible. 
  The actual directory location should be portable -- all paths should 
be relative, and you should be able to move the directory around.

lib/python2.4/ is for packages.  I'm almost inclined to say that 
--single-version-externally-managed makes sense on some level, with a 
record kept in some standard place (lib/python2.4/install-record.txt?) 
-- but I'm basically indifferent.  I at least don't see a need for 
multiple simultaneous versions in this setup, and multiple versions do 
lead to confusion.  Version metadata is still nice, of course.

src/ is for checkouts, where each package is installed with setup.py 
develop.  These are naturally single-version, which is part of why I 
like the idea of only using single-version setups.  I'm a little unsure 
of how src/ should be layed out.  In practice I want all "my" packages 
to be installed in src/ as checkouts, either from tags or the trunk (or 
a branch or whatever).  So I'm not sure if I should name the 
subdirectories after the package, or maybe even the package plus a tag 
name.

One of the things SwitchTower (now "Cappucino", I think) does in Rails 
land is it makes a dated checkout, then activates that checkout (it does 
it with a symlink, we'd do it with setup.py develop).  It then rolls 
back by switching to an existing checkout.  Of course svn switch + svn 
up does this in place, and with less checkout trash laying around, even 
if rollbacks aren't as fast as a result.  So, I'm thinking just 
src/PackageName/

There's an installation issue there -- it would be nice if I could say 
"these are the packages I want to install as editable" and easy_install 
would pick those up (maybe detecting based on what package index the 
package was found in) and install them in src/ as editable.

sys.path would contain /usr/lib/python2.4, optionally 
/usr/lib/python2.4/site-packages, and env/lib/python2.4/, and all the 
similar directories.  Unfortunately figuring out what "similar" 
directories there are is hard.  sys.path on my machine now has 63 
entries normally and 12 with python -S.  I guess I'd really like to 
start with 12 and build up, instead of 63 and try to strip them down.

Installation as a whole is an open issue.  Putting in env/setup.cfg with 
the setting specific to that working environment works to a degree -- 
easy_install will pick it up if invoked from there.  But that doesn't 
work with setup.py develop, or setup.py install, or some other 
scenarios.  The system distutils.cfg doesn't really work, because the 
only expansion it knows how to do is of user directories, so there's 
little way to pass interesting information in (like a "this is my 
setup.cfg" environmental variable or something).  Maybe with PYTHONPATH 
to indicate the working environment, and a distutils monkeypatch put 
into lib/python2.4/distutils/__init__.py?  I played around with putting 
the path setup in sitecustomize, but that runs after site.py, and 
doesn't run at all if python -S is used, so it seems like it brings in 
too much before it can remove stuff.

Another option is a completely new python interpreter bound to the 
environment.  Basically the virtual-python.py option 
(http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python). 
  In this model using env/bin/python indicate the proper environment, 
and you'd have local installs of *everything* including easy_install. 
This fixes so many problems without crazy hacks that it strongly appeals 
to me, especially if we can make it somewhat lighter.  I get this 
/usr/lib/python2.4.zip on my path, that doesn't usually exist (does it 
ever get created by default); if we could create that somehow on demand 
and use such a big-bundle-zip, that seems lighter and faster and nicer. 
  If we just put .pyc files in it, and those .pyc files refer back to 
the actual module source (in /usr/lib/python2.4/), then tracebacks 
should also still work, right?  No actual symlinks either, so it should 
work on Windows.  I'm not entirely sure where I'm going with this, though.

Sorry for the length.  I've been stewing on this without a lot of 
progress since PyCon so I thought I'd just throw out my current 
thoughts.  Maybe what I really want to do is hack on virtual-python.py 
some more and see where that gets me.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org