[Python-Dev] Proposal for virtualenv functionality in Python

[Larry Hastings]

Ian Bicking wrote:
> This is a proto-proposal for including some functionality from 
> virtualenv in Python itself.  I'm not entirely confident about what 
> I'm proposing, so it's not really PEP-ready, but I wanted to get 
> feedback...
>
> First, a bit about how virtualenv works (this will use Linux 
> conventions; Windows and some Mac installations are slightly different):
>
> * Let's say you are creating an environment in ~/env/
> * /usr/bin/python is *copied* to ~/env/bin/python
> * This alone sets sys.prefix to ~/env/ (via existing code in Python)
> * At this point things are broken because the standard library is not 
> available
> * virtualenv creates ~/env/lib/pythonX.Y/site.py, which adds the 
> system standard library location (/usr/lib/pythonX.Y) to sys.path
> * site.py itself requires several modules to work, and each of these 
> modules (from a pre-determined list of modules) is symlinked over from 
> the standard library into ~/env/lib/pythonX.Y/
> * site.py may or may not add /usr/lib/pythonX.Y/site-packages to sys.path
> * *Any* time you use ~/env/bin/python you'll get sys.prefix of ~/env/, 
> and the appropriate path.  No environmental variable is required.
> * No compiler is used; this is a fairly light tool

I have a tool that also creates a virtualized Python environment, but 
doesn't solve the problem as thoroughly as virtualenv.  I limp along by 
tweaking PYTHONUSERBASE and PYTHONPATH.  I'm very interested in seeing 
something like this make it in to Python.

A few inline comments:

> * I'd rather ~/env/bin/python be a symlink instead of copying it.

The thread discussing Windows suggests that we shouldn't use symlinks 
there.  I'd say either copying or symlinking pythonv should be 
supported, and on Windows we recommend copying pythonv.exe.


> * Compiling extensions can be tricky because code may not find headers 
> (because they are installed in /usr, not ~/env/).  I think this can be 
> handled better if virtualenv is slightly less intrusive, or distutils 
> is patched, or generally tools are more aware of this layout.

Conversely, headers may be installed in ~/env and not /usr.  The 
compiler should probably look in both places.  But IIUC telling the 
compiler how to do that is only vaguely standardized--Microsoft's CL.EXE 
doesn't seem to support any environment variable containing an include 
/path/.

I suspect solving this in a general way is out-of-band for pythonv, but 
I'm willing to have my mind changed.  Certainly pythonv should add its 
prefix directory to LD_LIBRARY_PATH on Linux.

> Additionally, the binary will look for a configuration file.  I'm not 
> sure where this file should go; perhaps directly alongside the binary, 
> or in some location based on sys.prefix.
>
> The configuration file would be a simple set of assignments; some I 
> might imagine:
>
> * Maybe override sys.prefix
> * Control if the global site-packages is placed on sys.path
> * On some operating systems there are other locations for packages 
> installed with the system packager; probably these should be possible 
> to enable or disable
> * Maybe control installations or point to a file like distutils.cfg

I'm unexcited by this; I think simpler is better.  pythonv should 
virtualize environments layered on top of python, and should have one 
obvious predictable behavior.  Certainly if it supports a configuration 
file pythonv should run without it and pick sensible defaults.

What are the use cases where you need these things to be configurable?


Let me propose further about python and pythonv:

    * As Antoine suggested, the CPython interpreter should sprout a new
      command-line switch, "--prefix", which adds a new prefix directory.
    * pythonv's purpose in life is to infer your prefix directory and
      run "pythonX.X --prefix <prefixdir> [ all args it got ... ]".
    * Should pythonv should be tied to the specific Python executable? 
      If you run install pythonv as "python", should it look for
      "python" or explicitly look for the specific Python it shipped
      with, like "python3.2"?  I suspect the latter though I'm no longer
      sure.


I'm one of those folks who'd like to see this be stackable.  If we tweak 
the semantics just a bit I think it works:

    * pythonv should inspect its --prefix arguments, as well as passing
      them on to the child python process it runs.
    * When pythonv wants to run the next python process in line, it
      scans the path looking for the pythonX.X interpreter but /ignores/
      all the interpreters that are in in a --prefix bin directory it's
      already seen.
    * python handles multiple --prefix options, and later ones take
      precedence over earlier ones.
    * What should sys.interpreter be?  Explicit is better than implicit:
      the first pythonv to run also adds a --interpreter <argv[0]> to
      the front of the command-line.  Or they could all add it and
      python only uses the last one.  This is one area where "python" vs
      "python3.2" makes things a little complicated.


I'm at PyCon and would be interested in debating / sprinting on this if 
there's interest.


/larry/