[Python-ideas] Draft PEP for virtualenv in the stdlib

[Carl Meyer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/26/2011 12:28 PM, Jim Jewett wrote:
> On Wed, Oct 26, 2011 at 1:06 PM, Carl Meyer <carl at oddbird.net> wrote:
>> On 10/26/2011 10:04 AM, Jim Jewett wrote:
> As best I can tell, the question is whether sys.prefix should point to
> the base system (because of header files?) or the virtual sandbox
> (because of user site packages?)

The standard library is also located always in the base system only.

> Will header files and such be explicitly hidden from applications
> running in the virtual environment?  It sounded like the answer was
> "depends on the flag value".  But it seems to me that:
> 
>     (a)  If the base system's data (such as headers) are explicitly hidden,
>     then there isn't much good reason to then say "oh, by the way, the
>     secret is hidden over here".
> 
>     (b)  If they are not explicitly hidden, then they (or more-correct
> overrides)
>     should also be available from the virtual environment (whether by copying
>     or by symlink, or even by symlink to the whole directory).

This dichotomy assumes an even more comprehensive form of isolation
(basically that which is provided already by PYTHONHOME). That level of
isolation is problematic precisely because it requires either copying or
symlinking the entire standard library into every environment
(essentially duplicating the base Python installation for every
environment, perhaps using symlinks to gain some efficiency). This is an
issue because:

1) Not all supported platforms have good support for symlinks, and
copying the entire standard library is quite heavyweight if you'll be
creating a lot of environments.

2) site-packages is contained within the standard library directory, and
you presumably want to override site-packages, so you can't just make a
single "lib" symlink to get the stdlib, you have to individually symlink
every single module.

If you aren't bothered by these caveats, then you can just use
PYTHONHOME today, and you have no need for this PEP.

The goal of this PEP is to provide site-packages isolation without
standard-library isolation (and its attendant downsides), by allowing
knowledge of the base system environment (enough to use its standard
library) even when Python is running within the virtual environment.

The Google code search linked in the PEP demonstrates that there is code
out there using sys.prefix to find the standard library (for instance,
to exclude it from tracing). This code will no longer function if
sys.prefix no longer points to the location where the standard library
is actually found.

It's possible that there is more code out there using sys.prefix to find
site-packages. However, the documented definition of sys.prefix never
mentions site-packages, whereas it does mention the standard library and
header files. So if we have to choose which code to break, I see a
strong argument for adhering to the documented definition rather than
breaking it.

(I could fairly easily be convinced, however, that practicality beats
purity in this case, especially since changing sys.prefix errs in the
direction of greater isolation.)

(It's worth noting that really no code outside the stdlib should be
using any of these sys prefix attributes directly, it should instead use
the appropriate APIs in site/sysconfig/distutils).

> So I'm still not seeing any situation where a normal program should
> care about the base value, but I am seeing plenty of situations where
> it might erroneously ask for that value and test (or install)
> something outside the virtual environment as a result.

The case you are missing is "any code which cares where the standard
library or header files are located and is using sys.prefix to figure
this out," because it is a design goal of this PEP to not have to
provide indirection for those things in the virtual environment layout.

Carl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6oV9UACgkQ8W4rlRKtE2eybQCgtEoLppBSj37rtnkqJ0aAkbq1
rjcAoJruTYLiw4fk/Pf1a68OpAnKhqzl
=MuWe
-----END PGP SIGNATURE-----