[Distutils] pythonv, take two

[Carl Meyer]

Hi Vinay,

On 03/29/2011 07:00 PM, Vinay Sajip wrote:
>> If we switch to sys.virtual_prefix and leave sys.prefix alone, the
>> danger would be that some installer that's using sys.prefix directly,
>> and ignoring sysconfig.py, would try to install stuff to the global
>> Python installation. But I'm not aware of any such installer, and I
>> think we could safely call any installer of Python code that ignores the
>> layout schemes in sysconfig thoroughly broken anyway.
>>
>> Your thoughts? (Or anyone else?)
> 
> What about the possibility of third-party code which uses
> sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.)

Right. The question is, for third-party code using
sys.prefix/sys.exec_prefix, which value would be the right one, the
installation prefix or the virtual prefix? It depends on what they're
using it for. I did a quick Google Code Search for "sys.prefix," and
unfortunately right on the front page I can find examples where either
approach we take would be "wrong."

The majority of uses I found are people copying code from trace.py which
is intended to exclude the standard library and system-wide packages
from coverage tracing. In this case, leaving sys.prefix pointing to the
installation prefix would give the desired result. Also, for instance,
here where matplotlib's setup looks for DLLs on Windows, the system
prefix would be correct: http://bit.ly/eQSVYc

On the other hand, here scons uses sys.prefix to try to find
site-packages: http://bit.ly/ik0PwJ  If we leave sys.prefix pointing to
the installation directory, this would leave scons unable to find its
config file in the virtualenv's site-packages directory. Unless they fix
their code to use the get_python_lib() function in sysconfig, which is
what they ought to be doing instead of duplicating all that logic.

So... it seems to me that we're likely to break _some_ third-party code
using sys.prefix regardless of what we do here. My instinct says adding
sys.virtual_prefix and leaving sys.prefix alone is the better approach,
but I'm not very firmly entrenched in that position.

> On the question of a C-readable configuration file, I've started work on this
> on the assumption that it's a UTF8-encoded file which contains lines of the
> form key = value. Having thought about it, since this file is only for
> bootstrapping the virtualenv anyway, there's no strong case for it to be
> ConfigParser-compatible.

Agreed. I only used ConfigParser initially because that was easiest from
Python code :-)

> I've modified calculate_path in getpath.c to look for the file adjacent to the
> executable or one level up. If found, we look for a line "home =
> /path/to/source/of/copied/python" and, if found, replace argv0_path with the
> value, then let the existing code take its course. I've modified your
> virtualize() in site.py to cater for the new config file format: we default the
> prefix to the parent of the directory containing the executable (hence, the
> virtualenv root), then set sys.virtual_prefix to this prefix, and optionally
> disallow the system site-packages from the system path. For now (since
> [distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set
> sys.prefix and sys.exec_prefix to sys.virtual_prefix.

This sounds good.

> So as not to cause confusion with existing config files you might have, I
> changed the config file name to 'env.cfg'. This can be changed to the
> appropriate bikeshed colour in two places - site.py and getpath.c.
> 
> Progress seems good. To test:
> 
> 1. Clone the repo at https://bitbucket.org/vinay.sajip/pythonv
> 2. In that repo, run ./configure and make
> 3. Download the "pmv.py" script at https://gist.github.com/893507
> 4. Run pmv.py with the python you just built - see the header comment in that
> script.
> 
> Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a
> copied rather than a symlinked Python.

Hmm, it doesn't seem to be working for me. sys.prefix and
sys.exec_prefix are still /usr/local using the python binary in my
"pmv", and sys.virtual_prefix is not set at all, despite env.cfg being
present and looking as it should.

Oddly when I run the compiled python binary directly from the checkout,
it does set sys.virtual_prefix to point one level above the checkout,
despite there being no env.cfg in the vicinity at all.

You can see both of these things (in reverse order) here:
http://paste.pocoo.org/show/362798/

I'll dig in a bit more and see if I can figure out what's happening.

Is there a reason you didn't base your changesets on mine (in the
cpythonv repo), and instead apparently copied over a bunch of the
changes manually as a diff? I can transplant or copy your work back, or
try to do a full merge, but in the long run if we'll both be working on
this it seems best if we are easily able to merge our work back and forth.

Carl