[Python-Dev] Proposal for virtualenv functionality in Python

[Ian Bicking]

This is a proto-proposal for including some functionality from virtualenv in
Python itself.  I'm not entirely confident about what I'm proposing, so it's
not really PEP-ready, but I wanted to get feedback...

First, a bit about how virtualenv works (this will use Linux conventions;
Windows and some Mac installations are slightly different):

* Let's say you are creating an environment in ~/env/
* /usr/bin/python is *copied* to ~/env/bin/python
* This alone sets sys.prefix to ~/env/ (via existing code in Python)
* At this point things are broken because the standard library is not
available
* virtualenv creates ~/env/lib/pythonX.Y/site.py, which adds the system
standard library location (/usr/lib/pythonX.Y) to sys.path
* site.py itself requires several modules to work, and each of these modules
(from a pre-determined list of modules) is symlinked over from the standard
library into ~/env/lib/pythonX.Y/
* site.py may or may not add /usr/lib/pythonX.Y/site-packages to sys.path
* *Any* time you use ~/env/bin/python you'll get sys.prefix of ~/env/, and
the appropriate path.  No environmental variable is required.
* No compiler is used; this is a fairly light tool

There are some tweaks to this that could be made, but I believe virtualenv
basically does things The Right Way.  By setting sys.prefix All Tools Work
(there are some virtualenv alternatives that do isolation without setting
sys.prefix, but they typically break more often than virtualenv, or only
support a limited number of workflows).  Also by using a distinct
interpreter (~/env/bin/python) it works fairly consistently and reliably
compared to techniques like an environmental variable.  The one serious
alternative is what buildout (and virtualenv --relocatable) does, which is
to use the system Python and change the path at the beginning of all scripts
(it requires its own installer to accomplish this consistently).

But virtualenv is kind of a hack, and I believe with a little support from
Python this could be avoided.  virtualenv can continue to exist to support
the equivalent workflows on earlier versions of Python, but it would not
exist (or would become much much simpler) on further Python versions.

The specific parts of virtualenv that are a hack that I would like to
replace with built-in functionality:

* I'd rather ~/env/bin/python be a symlink instead of copying it.
* I'd rather not copy (or symlink) *any* of the standard library.
* I'd rather site.py support this functionality natively (and in turn that
OS packagers support this when they make other modifications)
* Compiling extensions can be tricky because code may not find headers
(because they are installed in /usr, not ~/env/).  I think this can be
handled better if virtualenv is slightly less intrusive, or distutils is
patched, or generally tools are more aware of this layout.
* This gets more complicated with a Mac framework build of Python, and
hopefully those hacks could go away too.

I am not sure what the best way to do this is, but I will offer at least one
suggestion (other suggestions welcome):

In my (proto-)proposal, a new binary pythonv is created.  This is slightly
like pythonw.exe, which provides a Python interpreter on Windows which
doesn't open a new window.  This binary is primarily for creating new
environments.  It doesn't even need to be on $PATH, so it would be largely
invisible to people unless they use it.

If you symlink pythonv to a new location, it will effect sys.prefix
(currently sys.prefix is calculated after dereferencing the symlink).

Additionally, the binary will look for a configuration file.  I'm not sure
where this file should go; perhaps directly alongside the binary, or in some
location based on sys.prefix.

The configuration file would be a simple set of assignments; some I might
imagine:

* Maybe override sys.prefix
* Control if the global site-packages is placed on sys.path
* On some operating systems there are other locations for packages installed
with the system packager; probably these should be possible to enable or
disable
* Maybe control installations or point to a file like distutils.cfg

I got some feedback from the Debian/Ubuntu maintainer that he would like
functionality that might be like this; for instance, if you have
/usr/bin/python2.6 and /usr/bin/python2.6-dbg, he'd like them to work
slightly different (e.g., /usr/bin/python2.6-dbg would look in a different
place for libraries).  So the configuration file location should be based on
sys.prefix *and* the name of the binary itself (e.g.,
/usr/lib/python2.6/python-config-dbg.conf).  I have no strong opinion on the
location of the file itself, only that it can be specific to the directory
and name of the interpreter.

In addition to all this, I think sys would grow another prefixy value, e.g.,
sys.build_prefix, that points to the place where Python was actually built
(virtualenv calls this sys.real_prefix, but that's not a very good name).
 Some code, especially in distutils, might need to be aware of this to
compile extensions properly (we can be somewhat aware of these cases by
looking at places where virtualenv already has problems compiling
extensions).

Some people have argued for something like sys.prefixes, a list of locations
you might look at, which would allow a kind of nesting of these environments
(where sys.prefixes[-1] == sys.prefix; or maybe reversed).  Personally this
seems like it would be hard to keep mental track of this, but I can
understand the purpose -- you could for instance create a kind of template
prefix that has *most* of what you want installed in it, then create
sub-environments that contain for instance an actual application, or a
checkout (to test just one new piece of code).

I'm not sure how this should best work on Windows (without symlinks, and
where things generally work differently), but I would hope if this idea is
more visible that someone more opinionated than I would propose the
appropriate analog on Windows.


-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://twitter.com/ianbicking
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100219/49d99270/attachment.htm>