[Distutils] pythonv, take two

[Vinay Sajip]

Carl Meyer <carl <at> oddbird.net> writes:

> In general, I think the "copied binary" case should be as similar as
> possible to the "symlinked binary" case. Python doesn't dereference
> symlinks in setting sys.executable, so in either case sys.executable
> will point to the virtual environment's binary, which I think is clearly
> what we want (think code using sys.executable to decide how to re-exec
> itself).

I agree.

> The changes I made to sysconfig thus far were the minimum necessary to
> be able to successfully install stuff in a pythonv virtualenv. On a
> brief glance, it looks to me like you are right; we'll need to modify
> the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this
> is part -- maybe all -- of what's needed to fix issue #1).
> 
> Also, see issue #10 - I'm wavering on whether changing sys.prefix and
> sys.exec_prefix to point to the virtualenv is actually the right
> approach. We're already forced to modify sysconfig.py to be
> virtual-aware in order to avoid having to hackily copy/symlink bits of
> the stdlib and include directories and the Makefile and config.h and
> whatnot into the virtualenv. Once we've taken the step of making
> sysconfig virtual-aware, it seems to me that perhaps sys.prefix is
> better left pointing to the actual Python installation, and we add a
> sys.virtual_prefix or some such to tell site.py and sysconfig.py where
> "virtualized" stuff belongs.

This sounds better, too.

> If we switch to sys.virtual_prefix and leave sys.prefix alone, the
> danger would be that some installer that's using sys.prefix directly,
> and ignoring sysconfig.py, would try to install stuff to the global
> Python installation. But I'm not aware of any such installer, and I
> think we could safely call any installer of Python code that ignores the
> layout schemes in sysconfig thoroughly broken anyway.
> 
> Your thoughts? (Or anyone else?)

What about the possibility of third-party code which uses
sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.)

> P.S. The elephant in the room here is the fact that if we want pythonv
> to be backwards-compatible with distutils/distribute, which we do, we
> can't get away with just modifying sysconfig.py, we also have to modify
> the old distutils/sysconfig.py. In general distutils is frozen; I'm
> hoping that Tarek will let me get away with a few changes for pythonv if
> they aren't changing APIs at all, just modifying some install-scheme
> paths to be virtual-aware.

Here's hoping :-)

On the question of a C-readable configuration file, I've started work on this
on the assumption that it's a UTF8-encoded file which contains lines of the
form key = value. Having thought about it, since this file is only for
bootstrapping the virtualenv anyway, there's no strong case for it to be
ConfigParser-compatible.

I've modified calculate_path in getpath.c to look for the file adjacent to the
executable or one level up. If found, we look for a line "home =
/path/to/source/of/copied/python" and, if found, replace argv0_path with the
value, then let the existing code take its course. I've modified your
virtualize() in site.py to cater for the new config file format: we default the
prefix to the parent of the directory containing the executable (hence, the
virtualenv root), then set sys.virtual_prefix to this prefix, and optionally
disallow the system site-packages from the system path. For now (since
[distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set
sys.prefix and sys.exec_prefix to sys.virtual_prefix.

So as not to cause confusion with existing config files you might have, I
changed the config file name to 'env.cfg'. This can be changed to the
appropriate bikeshed colour in two places - site.py and getpath.c.

Progress seems good. To test:

1. Clone the repo at https://bitbucket.org/vinay.sajip/pythonv
2. In that repo, run ./configure and make
3. Download the "pmv.py" script at https://gist.github.com/893507
4. Run pmv.py with the python you just built - see the header comment in that
script.

Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a
copied rather than a symlinked Python.

Regards,

Vinay Sajip