[Distutils] pythonv, take two

[Carl Meyer]

Hi Vinay,

On 03/29/2011 09:45 AM, Vinay Sajip wrote:
> I've been thinking about the C-level configuration some more (cpythonv issue
> #6). Suppose we have a virtual env at absolute path <testenv>, and in
> <testenv>/bin we have a copied, non-installed Python, obtained from absolute
> path <pythonsrc>. Say we provide the location of the source in the
> configuration, and that this is read in getpath.c. Logically, it should just
> replace the argv0_path value, and if the subsequent logic is unchanged, at the
> point that site.py's main() runs,  you would have (on Linux i686):
> 
> sys.prefix, sys.exec_prefix would have values set from the configure script
> ('/usr/local' by default).
> sys.path would be ['/usr/local/lib/python33.zip', '<pythonsrc>/Lib',
> '<pythonsrc>/Lib/plat-linux2', '<pythonsrc>/build/lib.linux-i686-3.3']
> 
> which seems OK.
> 
> If a virtual environment is in absolute path <testenv>, then after the call to
> virtualize() in site.py, we would have sys.prefix and sys.exec_prefix both set
> to <testenv>. Since sys.executable still points to <testenv>/bin, the code in
> sysconfig.py appears not to be doing the right thing, e.g.
> sysconfig._PROJECT_BASE would be <testenv>/bin, which seems wrong.
> 
> So it seems as if site.py and/or sysconfig.py might need further changes, as
> well as changes to getpath.c. What do you think?

In general, I think the "copied binary" case should be as similar as
possible to the "symlinked binary" case. Python doesn't dereference
symlinks in setting sys.executable, so in either case sys.executable
will point to the virtual environment's binary, which I think is clearly
what we want (think code using sys.executable to decide how to re-exec
itself).

The changes I made to sysconfig thus far were the minimum necessary to
be able to successfully install stuff in a pythonv virtualenv. On a
brief glance, it looks to me like you are right; we'll need to modify
the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this
is part -- maybe all -- of what's needed to fix issue #1).

Also, see issue #10 - I'm wavering on whether changing sys.prefix and
sys.exec_prefix to point to the virtualenv is actually the right
approach. We're already forced to modify sysconfig.py to be
virtual-aware in order to avoid having to hackily copy/symlink bits of
the stdlib and include directories and the Makefile and config.h and
whatnot into the virtualenv. Once we've taken the step of making
sysconfig virtual-aware, it seems to me that perhaps sys.prefix is
better left pointing to the actual Python installation, and we add a
sys.virtual_prefix or some such to tell site.py and sysconfig.py where
"virtualized" stuff belongs.

There are a variety of possible meanings of sys.prefix, and the only one
we really want to modify is "where should I install third-party
packages?" Looking at sys.prefix references in the stdlib, I see them in
trace, tkinter, and idlelib. It appears to me that in all of those
cases, having sys.prefix pointing to the virtualenv would be wrong.

If we switch to sys.virtual_prefix and leave sys.prefix alone, the
danger would be that some installer that's using sys.prefix directly,
and ignoring sysconfig.py, would try to install stuff to the global
Python installation. But I'm not aware of any such installer, and I
think we could safely call any installer of Python code that ignores the
layout schemes in sysconfig thoroughly broken anyway.

Your thoughts? (Or anyone else?)

Carl

P.S. The elephant in the room here is the fact that if we want pythonv
to be backwards-compatible with distutils/distribute, which we do, we
can't get away with just modifying sysconfig.py, we also have to modify
the old distutils/sysconfig.py. In general distutils is frozen; I'm
hoping that Tarek will let me get away with a few changes for pythonv if
they aren't changing APIs at all, just modifying some install-scheme
paths to be virtual-aware.