Draft PEP for virtualenv in the stdlib
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Vinay Sajip and I are working on a PEP for making "virtual Python environments" a la virtualenv [1] a built-in feature of Python 3.3. This idea was first proposed on python-dev by Ian Bicking in February 2010 [2]. It was revived at PyCon 2011 and has seen discussion on distutils-sig [3] and more recently again on python-dev [4] [5]. Given all this (mostly positive) prior discussion, we may be at a point where further discussion should happen on python-dev rather than python-ideas. But in order to observe the proper PEP 1 process, I'm posting the draft PEP here first for pre-review and comment before I send it to the PEP editors and post it on python-dev. Full text of the draft PEP is pasted below, and also available on Bitbucket [6]. [1] http://virtualenv.org [2] http://mail.python.org/pipermail/python-dev/2010-February/097787.html [3] http://mail.python.org/pipermail/distutils-sig/2011-March/017498.html [4] http://mail.python.org/pipermail/python-dev/2011-June/111903.html [5] http://mail.python.org/pipermail/python-dev/2011-October/113883.html [6] https://bitbucket.org/carljm/pythonv-pep/src/ PEP: XXX Title: Python Virtual Environments Version: $Revision$ Last-Modified: $Date$ Author: Carl Meyer <carl@oddbird.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Jun-2011 Python-Version: 3.3 Post-History: 14-Jun-2011 Abstract ======== This PEP proposes to add to Python a mechanism for lightweight "virtual environments" with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in its site directories. Motivation ========== The utility of Python virtual environments has already been well established by the popularity of existing third-party virtual-environment tools, primarily Ian Bicking's `virtualenv`_. Virtual environments are already widely used for dependency management and isolation, ease of installing and using Python packages without system-administrator access, and automated testing of Python software across multiple Python versions, among other uses. Existing virtual environment tools suffer from lack of support from the behavior of Python itself. Tools such as `rvirtualenv`_, which do not copy the Python binary into the virtual environment, cannot provide reliable isolation from system site directories. Virtualenv, which does copy the Python binary, is forced to duplicate much of Python's ``site`` module and manually copy an ever-changing set of standard-library modules into the virtual environment in order to perform a delicate boot-strapping dance at every startup. The ``PYTHONHOME`` environment variable, Python's only existing built-in solution for virtual environments, requires copying the entire standard library into every environment; not a lightweight solution. A virtual environment mechanism integrated with Python and drawing on years of experience with existing third-party tools can be lower maintenance, more reliable, and more easily available to all Python users. .. _virtualenv: http://www.virtualenv.org .. _rvirtualenv: https://github.com/kvbik/rvirtualenv Specification ============= When the Python binary is executed, it attempts to determine its prefix (which it stores in ``sys.prefix``), which is then used to find the standard library and other key files, and by the ``site`` module to determine the location of the site-package directories. Currently the prefix is found (assuming ``PYTHONHOME`` is not set) by first walking up the filesystem tree looking for a marker file (``os.py``) that signifies the presence of the standard library, and if none is found, falling back to the build-time prefix hardcoded in the binary. This PEP proposes to add a new first step to this search. If an ``env.cfg`` file is found either adjacent to the Python executable, or one directory above it, this file is scanned for lines of the form ``key = value``. If a ``home`` key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the ``home`` key is the directory containing the Python executable used to create this virtual environment. In this case, prefix-finding continues as normal using the value of the ``home`` key as the effective Python binary location, which results in ``sys.prefix`` being set to the system installation prefix, while ``sys.site_prefix`` is set to the directory containing ``env.cfg``. (If ``env.cfg`` is not found or does not contain the ``home`` key, prefix-finding continues normally, and ``sys.site_prefix`` will be equal to ``sys.prefix``.) The ``site`` and ``sysconfig`` standard-library modules are modified such that site-package directories ("purelib" and "platlib", in ``sysconfig`` terms) are found relative to ``sys.site_prefix``, while other directories (the standard library, include files) are still found relative to ``sys.prefix``. Thus, a Python virtual environment in its simplest form would consist of nothing more than a copy of the Python binary accompanied by an ``env.cfg`` file and a site-packages directory. Since the ``env.cfg`` file can be located one directory above the executable, a typical virtual environment layout, mimicking a system install layout, might be:: env.cfg bin/python3 lib/python3.3/site-packages/ Isolation from system site-packages - ----------------------------------- In a virtual environment, the ``site`` module will normally still add the system site directories to ``sys.path`` after the virtual environment site directories. Thus system-installed packages will still be importable, but a package of the same name installed in the virtual environment will take precedence. If the ``env.cfg`` file also contains a key ``include-system-site`` with a value of ``false`` (not case sensitive), the ``site`` module will omit the system site directories entirely. This allows the virtual environment to be entirely isolated from system site-packages. Creating virtual environments - ----------------------------- This PEP also proposes adding a new ``venv`` module to the standard library which implements the creation of virtual environments. This module would typically be executed using the ``-m`` flag:: python3 -m venv /path/to/new/virtual/environment Running this command creates the target directory (creating any parent directories that don't exist already) and places an ``env.cfg`` file in it with a ``home`` key pointing to the Python installation the command was run from. It also creates a ``bin/`` (or ``Scripts`` on Windows) subdirectory containing a copy of the ``python3`` executable, and the ``pysetup3`` script from the ``packaging`` standard library module (to facilitate easy installation of packages from PyPI into the new virtualenv). And it creates an (initially empty) ``lib/pythonX.Y/site-packages`` subdirectory. If the target directory already exists an error will be raised, unless the ``--clear`` option was provided, in which case the target directory will be deleted and virtual environment creation will proceed as usual. If ``venv`` is run with the ``--no-site-packages`` option, the key ``include-system-site = false`` is also included in the created ``env.cfg`` file. Multiple paths can be given to ``venv``, in which case an identical virtualenv will be created, according to the given options, at each provided path. API - --- The high-level method described above will make use of a simple API which provides mechanisms for third-party virtual environment creators to customize environment creation according to their needs. The ``venv`` module will contain an ``EnvBuilder`` class which accepts the following keyword arguments on instantiation:: * ``nosite`` - A Boolean value indicating that isolation of the environment from the system Python is required (defaults to ``False``). * ``clear`` - A Boolean value which, if True, will delete any existing target directory instead of raising an exception (defaults to ``False``). The returned env-builder is an object which is expected to have a single method, ``create``, which takes as required argument the path (absolute or relative to the current directory) of the target directory which is to contain the virtual environment. The ``create`` method will either create the environment in the specified directory, or raise an appropriate exception. Creators of third-party virtual environment tools will be free to use the provided ``EnvBuilder`` class as a base class. The ``venv`` module will also provide a module-level function as a convenience:: def create(env_dir, nosite=False, clear=False): builder = EnvBuilder(nosite=nosite, clear=clear) builder.create(env_dir) The ``create`` method of the ``EnvBuilder`` class illustrates the hooks available for customization: def create(self, env_dir): """ Create a virtualized Python environment in a directory. :param env_dir: The target directory to create an environment in. """ env_dir = os.path.abspath(env_dir) context = self.create_directories(env_dir) self.create_configuration(context) self.setup_python(context) self.setup_packages(context) self.setup_scripts(context) Each of the methods ``create_directories``, ``create_configuration``, ``setup_python``, ``setup_packages`` and ``setup_scripts`` can be overridden. The functions of these methods are:: * ``create_directories`` - creates the environment directory and all necessary directories, and returns a context object. This is just a holder for attributes (such as paths), for use by the other methods. * ``create_configuration`` - creates the ``env.cfg`` configuration file in the environment. * ``setup_python`` - creates a copy of the Python executable (and, under Windows, DLLs) in the environment. * ``setup_packages`` - A placeholder method which can be overridden in third party implementations to pre-install packages in the virtual environment. * ``setup_scripts`` - A placeholder methd which can be overridden in third party implementations to pre-install scripts (such as activation and deactivation scripts) in the virtual environment. The ``DistributeEnvBuilder`` subclass in the reference implementation illustrates how these last two methods can be used in practice. It's not envisaged that ``DistributeEnvBuilder`` will be actually added to Python core, but it makes the reference implementation more immediately useful for testing and exploratory purposes. * The ``setup_packages`` method installs Distribute in the target environment. This is needed at the moment in order to actually install most packages in an environment, since most packages are not yet packaging / setup.cfg based. * The ``setup_scripts`` method installs activation and pysetup3 scripts in the environment. This is also done in a configurable way: A ``scripts`` property on the builder is expected to provide a buffer which is a base64-encoded zip file. The zip file contains directories "common", "linux2", "darwin", "win32", each containing scripts destined for the bin directory in the environment. The contents of "common" and the directory corresponding to ``sys.platform`` are copied after doing some text replacement of placeholders: * ``__VIRTUAL_ENV__`` is replaced with absolute path of the environment directory. * ``__VIRTUAL_PROMPT__`` is replaced with the environment prompt prefix. * ``__BIN_NAME__`` is replaced with the name of the bin directory. * ``__ENV_PYTHON__`` is replaced with the absolute path of the environment's executable. No doubt the process of PEP review will show up any customization requirements which have not yet been considered. Open Questions ============== Why not modify sys.prefix? - -------------------------- Any virtual environment tool along these lines is proposing a split between two different meanings (among others) that are currently both wrapped up in ``sys.prefix``: the answers to the questions "Where is the standard library?" and "Where is the site-packages location where third-party modules should be installed?" This split could be handled by introducing a new value for either the former question or the latter question. Either option potentially introduces some backwards-incompatibility with software written to assume the other meaning for ``sys.prefix``. Since it was unable to modify `distutils`, `virtualenv`_ has to re-point ``sys.prefix`` at the virtual environment, which requires that it also provide a symlink from inside the virtual environment to the Python header files, and that it copy some portions of the standard library into the virtual environment. The `documentation`__ for ``sys.prefix`` describes it as "A string giving the site-specific directory prefix where the platform independent Python files are installed," and specifically mentions the standard library and header files as found under ``sys.prefix``. It does not mention ``site-packages``. __ http://docs.python.org/dev/library/sys.html#sys.prefix It is more true to this documented definition of ``sys.prefix`` to leave it pointing to the system installation (which is where the standard library and header files are found), and introduce a new value in ``sys`` (``sys.site_prefix``) to point to the prefix for ``site-packages``. The justification for reversing this choice would be if it can be demonstrated that the bulk of third-party code referencing ``sys.prefix`` is, in fact, using it to find ``site-packages``, and not the standard library or header files or anything else. The most notable case is probably `setuptools`_ and its fork `distribute`_, which do use ``sys.prefix`` to build up a list of site directories for pre-flight checking where ``pth`` files can usefully be placed. It would be trivial to modify these tools (currently only `distribute`_ is Python 3 compatible) to check ``sys.site_prefix`` and fall back to ``sys.prefix`` if it doesn't exist. If Distribute is modified in this way and released before Python 3.3 is released with the ``venv`` module, there would be no likely reason for an older version of Distribute to ever be installed in a virtual environment. In terms of other third-party usage, a `Google Code Search`_ turns up what appears to be a roughly even mix of usage between packages using ``sys.prefix`` to build up a site-packages path and packages using it to e.g. eliminate the standard-library from code-execution tracing. Either choice that's made here will require one or the other of these uses to be updated. Another argument for reversing this choice and modifying ``sys.prefix`` to point at the virtual environment is that virtualenv currently does this, and it doesn't appear to have caused major problems. .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools .. _distribute: http://packages.python.org/distribute/ .. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs What about include files? - ------------------------- For example, ZeroMQ installs zmq.h and zmq_utils.h in $VE/include, whereas SIP (part of PyQt4) installs sip.h by default in $VE/include/pythonX.Y. With virtualenv, everything works because the PythonX.Y include is symlinked, so everything that's needed is in $VE/include. At the moment pythonv doesn't do anything with include files, besides creating the include directory; this might need to change, to copy/symlink $VE/include/pythonX.Y. I guess this would go into ``venv.py``. As in Python there's no abstraction for a site-specific include directory, other than for platform-specific stuff, then the user expectation would seem to be that all include files anyone could ever want should be found in one of just two locations, with sysconfig labels "include" & "platinclude". There's another issue: what if includes are Python-version-specific? For example, SIP installs by default into $VE/include/pythonX.Y rather than $VE/include, presumably because there's version-specific stuff in there - but even if that's not the case with SIP, it could be the case with some other package. And the problem that gives is that you can't just symlink the include/pythonX.Y directory, but actually have to provide a writable directory and symlink/copy the contents from the system include/pythonX.Y. Of course this is not hard to do, but it does seem inelegant. OTOH it's really because there's no supporting concept in Python/sysconfig. Interface with packaging tools - ------------------------------ Some work will be needed in packaging tools (Python 3.3 packaging, Distribute) to support implementation of this PEP. For example: * How Distribute and packaging use sys.prefix and/or sys.site_prefix. Clearly, in practice we'll need to use Distribute for a while, until packages have migrated over to usage of setup.cfg. * How packaging and Distribute set up shebang lines in scripts which they install in virtual environments. Add a script? - ------------- Perhaps a ``pyvenv`` script should be added as a more convienent and discoverable alternative to ``python -m venv``. Testability and Source Build Issues - ----------------------------------- In order to be able to test the ``venv`` module in the Python regression test suite, some anomalies in how sysconfig data is configured in source builds will need to be removed. For example, sysconfig.get_paths() in a source build gives (partial output): { 'include': '/home/vinay/tools/pythonv/Include', 'libdir': '/usr/lib ; or /usr/lib64 on a multilib system', 'platinclude': '/home/vinay/tools/pythonv', 'platlib': '/usr/local/lib/python3.3/site-packages', 'platstdlib': '/usr/local/lib/python3.3', 'purelib': '/usr/local/lib/python3.3/site-packages', 'stdlib': '/usr/local/lib/python3.3' } Activation and Utility Scripts - ------------------------------ Virtualenv currently provides shell "activation" scripts as a user convenience, to put the virtual environment's Python binary first on the shell PATH. This is a maintenance burden, as separate activation scripts need to be provided and maintained for every supported shell. For this reason, this PEP proposes to leave such scripts to be provided by third-party extensions; virtual environments created by the core functionality would be used by directly invoking the environment's Python binary. If we are going to rely on external code to provide these conveniences, we need to check with existing third-party projects in this space (virtualenv, zc.buildout) and ensure that the proposed API meets their needs. (Virtualenv would be fine with the proposed API; it would become a relatively thin wrapper with a subclass of the env builder that adds shell activation and automatic installation of ``pip`` inside the virtual environment). Ensuring that sys.site_prefix and sys.site_exec_prefix are always set? - ---------------------------------------------------------------------- Currently the reference implementation's modifications to standard library code use the idiom ``getattr(sys, "site_prefix", sys.prefix)``. Do we want this to be the long-term pattern, or should the sys module ensure that the ``site_*`` attributes are always set to something (by default the same as the regular prefix attributes), even if ``site.py`` does not run? Reference Implementation ======================== The in-progress reference implementation is found in `a clone of the CPython Mercurial repository`_. To test it, build and install it (the virtual environment tool currently does not run from a source tree). - From the installed Python, run ``bin/python3 -m venv /path/to/new/virtualenv`` to create a virtual environment. The reference implementation (like this PEP!) is a work in progress. .. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv References ========== Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6lrJMACgkQ8W4rlRKtE2dz4wCgqxtiHQr3ZEH/s1h069e15bu7 c70AoOSTd7drIp1g6z2QiuDKoTok6TRw =9XEL -----END PGP SIGNATURE-----
On Tue, Oct 25, 2011 at 4:21 AM, Carl Meyer <carl@oddbird.net> wrote:
The repeated references to copying binaries and Python files throughout the PEP is annoying, and needs to be justified. Python 3.2+ supports symlinking on Windows Vista and above as well as on *nix systems, so there needs to be a short section somewhere explaining why symlinks are not an adequate lightweight solution (pointing out the fact that creating symlinks on Windows often requires administrator privileges would be sufficient justification for me). (Obviously, 3rd party virtual environment solutions generally *do* have to copy things around, since Python 2.x doesn't expose symlink support on Windows at all)
It can also take advantage of the native symlink support to minimise copying.
Currently, the PEP uses a mish-mash of 'env', 'venv', 'pyvenv' and 'pythonv' and 'site' to refer to different aspects of the proposed feature. I suggest standardising on 'venv' wherever the Python relationship is implied, and 'pyvenv' wherever the Python relationship needs to be made explicit. So I think the name of the configuration file should be "pyvenv.cfg" (tangent: 'setup' lost its Python connection when it went from 'setup.py' to 'setup.cfg'. I wish the latter has been called 'pysetup.cfg' instead)
'site' is *way* too overloaded already, let's not make it worse. I suggest "sys.venv_prefix".
The builtin virtual environment mechanism should be specified to symlink things by default, and only copy things if the user specifically requests it. System administrators rightly fear the proliferation of multiple copies of binaries, since it can cause major hassles when it comes time to install security updates.
"site" is ambiguous here - rather than abbreviating, I suggest making the option "include-system-site-packages".
As noted above, those should be symlinks rather than copies, with copying behaviour explicitly requested via a command line option. Also, why "Scripts" rather than "bin" on Windows? The Python binary isn't a script. I'm actually not seeing the rationale for the obfuscated FHS inspired layout in the first place - why not dump the binaries adjacent to the config file, with a simple "site-packages" directory immediately below that? If there are reasons for a more complex default layout, they need to be articulated in the PEP. If the problem is wanting to allow cross platform computation of things like the site-packages directory location and other paths, then the answer to that seems to lie in better helper methods (whether in sysconfig, site, venv or elsewhere) rather than Linux specific layouts inside language level virtual environments.
Yikes, double negatives in APIs are bad news (especially when the corresponding config file option is expressed positively) I suggest this parameter should be declared as "system_site_packages=True".
In line with my above comments, I think there should be a third parameter here declared as "use_symlinks=True". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Oct 25, 2011 at 8:31 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
It's justification for supporting it, not necessarily for doing it implicitly (although Windows in general is far more tolerant of bundling than the various *nix platforms). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Oct 25, 2011 at 6:11 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Yeah, I realised I don't actually mind if things get copied around on Windows - it's the POSIX systems where implicit copying would bother me, and that goes to the heart of a longstanding difference in packaging philosophy between the two platforms :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10/25/2011 4:28 AM, Nick Coghlan wrote:
I have a different issue with copying. I have a new Win7 system with a solid state disk for programs (that usually sit unchanged months or years at a time). It is nice and fast, but has a finite write-cycle lifetime. So unnecessary copies are not nice. I presume venv could be told to copy onto the data disk, but then everything runs slower. -- Terry Jan Reedy
On Oct 26, 11:26 am, Terry Reedy <tjre...@udel.edu> wrote:
As you said, though, you're using it for primarily static programs. If you're concerned with its lifetime, would you really use it for development? The onus that SSD afficiendos place on others to not use drives as writeable data stores seems absolutely crazy to me.
On 10/25/2011 11:24 PM, alex23 wrote:
I don't. I put a (static) .pth file in site-packages that points to my development directory on the hard disk, where I happily edit and test sometimes several times an hour.
The onus that SSD afficiendos place on others to not use drives as writeable data stores seems absolutely crazy to me.
I said a) I would prefer to not have an unnecessary duplication but b) if a copy is to be made, I would like to have a choice of which drive to use. That does not strike me as 'absolutely crazy'. -- Terry Jan Reedy
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Terry, On 10/25/2011 10:50 PM, Terry Reedy wrote:
As things stand with the PEP, you can create virtualenvs wherever you like, and you can optionally use symlinks if you prefer. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nmegACgkQ8W4rlRKtE2dMwQCfevz5GgWK0nl3O4vqegsk902/ JTwAn2yu/MAb9a8nAYk0SJMfO/b32Yih =C3jN -----END PGP SIGNATURE-----
On 26 October 2011 14:24, alex23 <wuwei23@gmail.com> wrote:
As such an afficionado, I agree that trying not to use it as a writeable data store seems crazy. All current SSDs have a sufficient number of write-erase cycles that you would need to be writing to the entire drive non-stop for years before it became an issue. If you're worried, get more RAM and configure a RAM disk as your temporary store. If nothing else, it speeds up compiles significantly ... Tim Delaney
On Tue, 25 Oct 2011 21:26:07 -0400 Terry Reedy <tjreedy@udel.edu> wrote:
Uh, please do not throw around arguments like that without backing them up with concrete numbers. In practice, your SSD will have amply enough write-cycles for venv to work fine. And if it doesn't, you've simply been swindled. Regards Antoine.
----- Original Message -----
I agree that symlinking should be done wherever possible, but let's remember that it's not just a Python issue: Windows XP does not support true symlinks, but only "junctions" aka "reparse points". Of course, that's no reason not to use true symlinks where they *are* available.
On Mon, Oct 24, 2011 at 20:16, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
On 3.2+, symlinks on Windows are possible but only in what I suspect is a pretty rare version and environment combination. As said earlier, they work on Vista and beyond, but they require that the process has been elevated in order to obtain the symlink privilege. For example, even as an administrator account on Windows 7, you need to explicitly open up cmd with "Run as Administrator" to properly symlink (through Python or otherwise). If we can't execute the symlink, we raise OSError. We might as well try symlinks then fallback on OSError.
On Tue, Oct 25, 2011 at 11:42 AM, Brian Curtin <brian.curtin@gmail.com> wrote:
I'd prefer some kind of warning if the symlink fails on a POSIX system. On Windows, bundling is such an accepted practice that even copying by default wouldn't bother me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Vinay Sajip wrote:
Windows XP does not support true symlinks, but only "junctions" aka "reparse points".
Out of curiosity, how far do these fall short of being true symlinks? The points I'm aware of are: * They only work within a volume * The GUI doesn't know about them, so it's easy to mistake a link to a folder for an independent copy of it and accidentally trash the original Are there any others? -- Greg
On 25 October 2011 16:16, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Junctions are only available for directories, and they're more akin in practice to hardlinks than symlinks (although different to both). Junctions can quite happily work cross-volume. It's hardlinks that only work within a volume (and you can't make a hardlink to a directory). Junctions are transparent within a network share i.e. if you a share that contains a junction, the junction will be traversed correctly. Symlinks do not work as you would expect when accessed via a network share. On Win7 and Vista, deleting a junction only deletes the junction. On XP deleting a junction deletes the contents as well unless Link Shell Extension <http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html> is installed. Tim Delaney
On 25 October 2011 06:16, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
They are very uncommon on Windows in general, so people find them confusing when they encounter them (for example, as a consequence of the GUI issue mentioned above - I know that's happened to me). Is there any reason that hard links can't be used on Windows? (Still not cross-volume, still relatively rarely used, but a bit better known than symlinks). BTW, regardless of whether symlinks, hardlinks or copies end up being used, I'm +1 on the general proposal. Paul.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Nick, Thanks for the feedback, replies below. On 10/24/2011 04:11 PM, Nick Coghlan wrote:
Do you mean pointing out why symlinks are not in themselves an adequate solution for virtual Python environments, or why they aren't used in this implementation? I've updated the introductory paragraph you quoted to provide a bit more detail on the first question. I think the answer to the latter question is just that we were trying to keep things simple and consistent, and make it work the same way on as wide a variety of platforms as possible (e.g. Windows XP). Also, in earlier discussions on distutils-sig some people considered it a _feature_ to have the virtual environment's Python binary copied, making the virtual environment more isolated from system changes. Obviously, this is an area where programmers and sysadmins often don't see eye to eye ;) The technique in this PEP works just as well with a symlinked binary, though, and I don't see much reason not to provide a symlink option. Whether it is on by default where supported is something that may need more discussion (I don't personally have a strong opinion either way). The reason virtualenv copies rather than symlinks the binary has nothing to do with lack of symlink support in Python 2, it's because getpath.c dereferences a symlinked binary in finding sys.prefix, so virtualenv's isolation technique simply doesn't work at all with a symlinked binary. Our version with the config file and Vinay's changes in getpath.c does work with a symlinked binary.
I don't think this is a significant enough difference to warrant mention here. Existing virtualenv on Python 2 already can and does use symlinks (for the bits of the stdlib it needs) on platforms that have os.symlink. IOW, the details and supported platforms differ, but on both Python 2 and 3 the best you can do is try to use os.symlink where available, and be prepared to fall back to a copy.
Good point. We were already attempting to standardize just as you suggest, but hadn't renamed "env.cfg", and I missed one remaining instance of "pythonv" in the text (it's also used for legacy reasons in the reference implementation bitbucket repo name, but that doesn't seem worth changing).
This makes sense to me. I've updated the PEP accordingly and created an issue to remind me to update the reference implementation as well.
'site' is *way* too overloaded already, let's not make it worse. I suggest "sys.venv_prefix".
My original thinking here was that sys.site_prefix is an attribute that should always exist, and always point to "where stuff should be installed to site-packages", whether or not you are in a venv (if you are not, it would have the same value as sys.prefix). It's a little odd to use an attribute named "sys.venv_prefix" in that way, even if your code doesn't know or care whether its actually in a venv (and in general we should be encouraging code that doesn't know or care). (The attribute doesn't currently always-exist in the reference implementation, but I'd like to change that). I agree that "site" is overloaded, though. Any ideas for a name that doesn't further overload that term, but still communicates "this attribute is a standard part of Python that always has the same meaning whether or not you are currently in a venv"?
To be clear, "things" here refers only to the Python binary itself. The only other things that might be installed in a new environment are scripts (e.g. pysetup3), and those must be created anew, neither symlinked nor copied, as their shebang line needs to point to the venv's Python (or more complicated chicanery with .exe wrappers on Windows, unless we get PEP 397 in time).
I think both options should be allowed, and I don't have a strong feeling about the default.
Ok - updated the draft PEP, will update reference implementation.
Also, why "Scripts" rather than "bin" on Windows? The Python binary isn't a script.
No, but in real-world usage, scripts from installed packages in the virtualenv will be installed there. Putting the Python binary in the same location as the destination for installed scripts is a pretty important convenience, as it means you only need to add a single directory to the beginning of your shell path to effectively "activate" the venv.
The historical reason is that it emulates the layout found under sys.prefix in a regular Python installation (note that it's not actually Linux-specific, in virtualenv it matches the appropriate platform; i.e. on Windows it's "Lib\" rather than "lib\pythonX.X"). This was necessary for virtualenv because it couldn't make changes in distutils/sysconfig). I think there may be good reason to continue to follow this approach, simply because it makes the necessary changes to distutils/sysconfig less invasive, reducing the need for special-casing of the venv case. But I do need to look into this a bit more and update the PEP with further rationale in any case. Regardless, I would not be in favor of dumping binaries directly next to pyvenv.cfg. It feels cleaner to keep scripts and binaries in a directory specifically named and intended for that purpose, which can be added to the shell PATH. I also think there is some value, all else being roughly equal, in maintaining consistency with virtualenv's layout. This is not an overriding concern, but it will make a big difference in how much existing code that deals with virtual environments has to change.
Fair enough, updated in draft PEP.
Thanks again for the review! The updated draft is available on Bitbucket [1], and the open issues for the reference implementation (which should reflect outstanding differences from the draft PEP) are as well [2]. Carl [1] https://bitbucket.org/carljm/pythonv-pep/src [2] https://bitbucket.org/vinay.sajip/pythonv/issues?status=new&status=open -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6m8+gACgkQ8W4rlRKtE2f1EgCfZNLBXSI08UQdLCRQMYwxwAp3 ByoAn3cVYvQXWMc1xkoO6mMSmNBQbEAD =FzBA -----END PGP SIGNATURE-----
On 25 October 2011 18:37, Carl Meyer <carl@oddbird.net> wrote:
One thought - on "per user" Windows installations, or uninstalled Python builds on Windows, IIRC python.exe requires pythonXY.dll to be alongside the EXE. You should probably consider copying/linking that if it's not installed systemwide. It's not entirely obvious to me how you'd detect this, though - maybe take the simple approach of copying python.exe, and also copying pythonXY.dll if it's in the same directory otherwise assume it's a system install. There's also pythonw.exe on Windows, of course... (I'm assuming that w9xpopen.exe is probably not needed - I'm not even sure there's any remaining need to ship it with Python in the first place). Paul.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Paul, On 10/25/2011 12:11 PM, Paul Moore wrote:
Actually, the reference implementation does symlink/copy all DLLs from the same directory as the python binary or a DLLs/ subdirectory, and also any executable beginning with "python", which would include pythonw.exe. We should certainly update the PEP with more detail about how things look different on Windows, currently it's pretty much describing the POSIX behavior (my fault; Vinay's done all the Windows work and I'm not as familiar with it). Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nAjQACgkQ8W4rlRKtE2dF4gCfaBpwyQArhArI6ybo9c6Qbjsz mekAn3/LnNHnt3U79ZA6GIyZXxxE4Ja2 =whkG -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Oct 25, 2011, at 11:37 AM, Carl Meyer wrote:
I definitely think folks will want copy and symlink options, for exactly the reason you and Nick point out. As for what the default should be, I think maximal isolation probably wins, and thus copying should be the default. I wouldn't scream if it were the other way 'round though.
That makes sense to me. It would be nice to have the location of your site-prefix always available for calculations regardless of your "virtualness". Note that in all likelihood, Debian will have to modify that so that it points to dist-packages instead of site-packages. Will there be an easy hook/config file or some such that we could use to make that change?
sys.addons_directory, sys.plugins_directory, sys.external_packages or some such?
+1
All of that makes sense to me, so adding these rationales to the PEP would be useful. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJOpx1XAAoJEBJutWOnSwa/I2YP/jXyJCu5uIVp20H+SjlqlnmB y5tbzgSTPi3Yf1In1+bs1bPdNhqlZYi1WfmGRPA7QfXWK4oeSpPXTjs/bAaTYmPE GjWDvQKue1hNv8qdFbo0IyI3u+2Zq0KB3sC+ACxB8WZ/Xc9wt4q47zMOTods4/Zj ct5nvsib6xF/ZGk2BJopEQwFHhraxuNkrhSsgbUUsQJ3XEmVtLQZ4Dyr5gxRDmOF /oShU556C3JjyPsXkNmURjQgXsbsWN/7j815yVpsLWNm5fjCPjqJAjY64RBFe9l/ EJqutLvqfQJJ1ie/R8xBBLF1LWNzzOeWgcgenbrdlgXa7xuGQE8psb1CSxdEUgec 9CUqsZbXDhQDtgdIcqsemhtGSIrl8UtCyrIDj7HUqK6ibOLTCa83AztpyupXpcki ynTwJLdSYRc+8IH6kLQ5DpqYCYC+14jEoMrDSSRGB/EyAE1P0l3DE710JtYHV12R 6e0Ciw8qjuhUTnmAr6H5DnT8UaWXENBDkDbel6ABHxEKalavAnr3kzjUTLcnTwMJ QfrbcVDBRnGV1swelLZ3oXitgofCCaz7Ca4kwqHikUQ5R/iPNl5AJyyJenFHsPrc J0jfSQEqPP3FOO27atPlVy/9lE45Ac0D7UcMq5IrfruLd4Rw7P50sPIxa+anAfKK GEuw81sCOyyJ4PU0cfxw =KObT -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Barry, Thanks for the review and feedback. On 10/25/2011 02:34 PM, Barry Warsaw wrote:
As it currently stands, this is really just the/a _prefix_ under which site-packages is found (parallel to how sys.prefix is currently used). Which means "site-packages" vs "dist-packages" is still determined in site.py, which tacks platform-specific paths onto the end of the prefix in order to find the actual site-packages directory. So in the Debian case, you'd still have to modify your distributed site.py (and take a bit of care, assisted by the tests, that you don't break venv.py in the process); I don't think this PEP really changes that much. (Though the fact that not breaking virtual environments when you make distro-specific modifications to site.py becomes your responsibility as the distro Python maintainer rather than mine as virtualenv maintainer is, in my humble opinion, one of the greatest advantages of this PEP <wink>) Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nYLEACgkQ8W4rlRKtE2ftGgCfQr0m9WM5BtDxFZfqepbxqqxn +kIAniTBdvxWh/Bsg4rOEdGMYnIIpKmD =CMax -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Oct 25, 2011, at 07:21 PM, Carl Meyer wrote:
Yes, fair enough! :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJOp2b5AAoJEBJutWOnSwa/irAQAMN+ZtkWAVRdcs9XosYyWsaT gs/BiAMJ2pob0WZ+U0/8BDM5/uz7UYoPqzmn6EzDKKA+ezhJ8UNYwWTkQoUpuQK3 OqTAA06qJ3oVR+BYoC8kzriTWNxuJ6jCdJbRrK8PXTz3FBdhw720mHO1A5k3ZmEA 2HebsbO7X11iU5JyNV4+K98DzUna/RnLvT3RUm8NPjUpBIS5QHhnVpmFEHrEy1F0 QnyZXbY7UqpUxXtBY6Q1sHoERRmZ2EZOeHwzBMC0iUcig3Pmps+taxHqXxhPh72F cNSt76K/ceHlNQrB8pBKQnP2FUn5KrZ8ut+LNHQmP6LYSZ/2r0kExbbqzF7xIxce ACfIl1H/ezzA7KK29WlKMgqMU6K/OL2pcdtZry35wKCA1RIghpds44iEV26yqWPK EY428lmWrkPZ2YGsfGyrm5E5vOzWEmcp7ze9FY34vgiphMzAkl1opCqKRffSc+ci dzmmjAOot+J6jVX0zAPqwI03Uys8/HJZwY8kK5VyKJon5EVI5FI9C/gqBDwR/2vP QZzdnlswwgGSz9xclzOhPW66lCZ5duNC115evNBQ030zoQRnGNICc+SNRB2GYY2v gzutj9KT2sB+HuvQagQfk8zuLs5pCSd9jLxXb5JPxLb8NCJsxHG8LrZfm+HiVD58 QOUkGdfxEDv76AD4f6mu =psL3 -----END PGP SIGNATURE-----
On Wed, Oct 26, 2011 at 3:37 AM, Carl Meyer <carl@oddbird.net> wrote:
I've been persauded that copying by default is reasonable - so long as the option to use symlinks is there, sysadmins can put a policy for their organisation in place that requires its use.
I'd actually prefer that we use the explicit "sys.prefix" and "sys.venv_prefix" naming (with the latter set to None when not in a virtual env) and possibly offer a convenience API somewhere that hides the "sys.prefix if sys.venv_prefix is None else sys.venv_prefix" dance. There's already a site.getsitepackage() API that gets the full list of site package directories (it's not a given that there's only one).
As Barry said, the rationale sounds reasonable, but the PEP needs to explain it in terms those of us not fully versed in the details of cross-platform virtual environments can understand (examples of both *nix and Windows layouts with a simple example package installed would probably be useful on that front) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Oct 26, 2011 at 11:51 AM, Barry Warsaw <barry@python.org> wrote:
Yeah, having venv_prefix == prefix in the "not in a virtual env case" is fine by me as well. I think Carl's right that it reads a little oddly sometimes, but it's a better option than: - further overloading "site" (when more than site-package may be located using the venv prefix) - requiring people to fall back to sys.prefix explicitly The "am I in a virtual env?" check can then just be "if sys.prefix == sys.venv_prefix". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/25/2011 08:05 PM, Nick Coghlan wrote:
What about "sys.local_prefix"? Doesn't overload site, but also doesn't imply that it always points to a venv. I think "local" carries the right connotation here - this is the prefix for the user's local environment (as possibly opposed to the "global" system environment). Seems like this might make code using the attribute unconditionally read a bit less oddly?
The "am I in a virtual env?" check can then just be "if sys.prefix == sys.venv_prefix".
Right; though in general the goal is that by simply switching to use the new prefix in the right places in sysconfig, such an explicit "am in a venv" check should never be necessary, even in the stdlib. I think the reference implementation largely achieves this goal, I'd have to check again to see if that's entirely true. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nbPMACgkQ8W4rlRKtE2dbrQCg3iwUkXZUBHerzyBFq+jOWS91 pgkAn2hYy7pREwdFBQQayR2OLP9x62e9 =tx9A -----END PGP SIGNATURE-----
On Wed, Oct 26, 2011 at 12:14 PM, Carl Meyer <carl@oddbird.net> wrote:
I think explaining to people "sys.venv_prefix is still valid when you're not in a virtual env, it just points to the same place as sys.prefix" is easier than explaining a *new* name with "<whatever name> points to the virtual env directory when you're in a virtual environment and sys.prefix otherwise". So while I agree your concern is valid, I think just living with the quirkiness is a reasonable approach. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan@...> writes:
I thought Carl was saying that we use "local_prefix" as opposed to "venv_prefix", because the "local_" prefix seems logical and doesn't override "site". In a venv, local_prefix/local_exec_prefix point to the venv, otherwise they have the same values as prefix/exec_prefix. The "venv_XXX" does grate a bit, especially as we're trying to achieve a don't-know-or-care-if- I'm-in-a-venv as much as possible. And thinking more about overriding "site" - what is a "site" anyway? It seems to be a combination of Python version, standard library and a specific set of packages (comprising in the general case include files, shared libs and extension modules) - if I installed 3 different versions of Python on my system, I would have three different sites. By that definition, a venv is really a site, albeit a pared-down one which references shared code and library where possible. So site_prefix/site_exec_prefix don't seem unreasonable to me, or failing that, local_prefix/local_exec_prefix would be preferable to venv_prefix/venv_exec_prefix. Regards, Vinay Sajip
On Oct 26, 2011, at 09:02 AM, Vinay Sajip wrote:
I agree in general with these observations. What we're currently calling "site" feels more like "system" to me, although we probably can't change that until Python 4000. One other reason I like "local" over "venv" is that "venv" isn't actually a word. While it'll probably make it easier to google, it looks fairly jarring to me. In general I try to avoid made up words and abbreviations because they are harder for non-native English speakers to use (so I've been told by some non-native English speakers). -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Nick, On 10/25/2011 08:26 PM, Nick Coghlan wrote:
FWIW, I agree with Vinay and Barry here. It's not something I care deeply about (since I think direct use of these sys attributes should be discouraged outside the stdlib anyway), but I don't agree that a misnamed attribute is easier to explain than a new name. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oTYMACgkQ8W4rlRKtE2dXbgCggLUDFr1cqRrOTlvPH9PoC0y5 Y7QAoMyQtRh22P2ly7TmrS3/SlQncFFd =swWE -----END PGP SIGNATURE-----
On Tue, Oct 25, 2011 at 9:15 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Nick Coghlan <ncoghlan@...> writes:
But why is that better than a site.venv_prefix which points to a venv if you're in one, and == sys.prefix if you're not?
Is there a reason we can't just make sys.* be the virtual environment's version, and sys.base.* be the site-wide version? You could still find sys.base.*, and you could still check whether sys.prefix == sys.base.prefix, but programs that don't get updated will use the sandboxed virtual version by default. -jJ
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jim, On 10/26/2011 10:04 AM, Jim Jewett wrote:
Yes, this is a reasonable alternative that Vinay and I discussed at some length earlier in development. There are reasonable arguments to be made in both directions - you can read my summary of those arguments in the "Open Questions" section of the PEP. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oPh4ACgkQ8W4rlRKtE2ftFwCgmmTHdduy6C4f8fihMs2Q0/Nw E18AoMAExZGageDgI+1BzikBsrzQCYd+ =+bJn -----END PGP SIGNATURE-----
On Wed, Oct 26, 2011 at 1:06 PM, Carl Meyer <carl@oddbird.net> wrote:
On 10/26/2011 10:04 AM, Jim Jewett wrote:
Even after re-reading, I'm not sure I fully understand. As best I can tell, the question is whether sys.prefix should point to the base system (because of header files?) or the virtual sandbox (because of user site packages?) Will header files and such be explicitly hidden from applications running in the virtual environment? It sounded like the answer was "depends on the flag value". But it seems to me that: (a) If the base system's data (such as headers) are explicitly hidden, then there isn't much good reason to then say "oh, by the way, the secret is hidden over here". (b) If they are not explicitly hidden, then they (or more-correct overrides) should also be available from the virtual environment (whether by copying or by symlink, or even by symlink to the whole directory). So I'm still not seeing any situation where a normal program should care about the base value, but I am seeing plenty of situations where it might erroneously ask for that value and test (or install) something outside the virtual environment as a result. -jJ
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/26/2011 12:28 PM, Jim Jewett wrote:
The standard library is also located always in the base system only.
This dichotomy assumes an even more comprehensive form of isolation (basically that which is provided already by PYTHONHOME). That level of isolation is problematic precisely because it requires either copying or symlinking the entire standard library into every environment (essentially duplicating the base Python installation for every environment, perhaps using symlinks to gain some efficiency). This is an issue because: 1) Not all supported platforms have good support for symlinks, and copying the entire standard library is quite heavyweight if you'll be creating a lot of environments. 2) site-packages is contained within the standard library directory, and you presumably want to override site-packages, so you can't just make a single "lib" symlink to get the stdlib, you have to individually symlink every single module. If you aren't bothered by these caveats, then you can just use PYTHONHOME today, and you have no need for this PEP. The goal of this PEP is to provide site-packages isolation without standard-library isolation (and its attendant downsides), by allowing knowledge of the base system environment (enough to use its standard library) even when Python is running within the virtual environment. The Google code search linked in the PEP demonstrates that there is code out there using sys.prefix to find the standard library (for instance, to exclude it from tracing). This code will no longer function if sys.prefix no longer points to the location where the standard library is actually found. It's possible that there is more code out there using sys.prefix to find site-packages. However, the documented definition of sys.prefix never mentions site-packages, whereas it does mention the standard library and header files. So if we have to choose which code to break, I see a strong argument for adhering to the documented definition rather than breaking it. (I could fairly easily be convinced, however, that practicality beats purity in this case, especially since changing sys.prefix errs in the direction of greater isolation.) (It's worth noting that really no code outside the stdlib should be using any of these sys prefix attributes directly, it should instead use the appropriate APIs in site/sysconfig/distutils).
The case you are missing is "any code which cares where the standard library or header files are located and is using sys.prefix to figure this out," because it is a design goal of this PEP to not have to provide indirection for those things in the virtual environment layout. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oV9UACgkQ8W4rlRKtE2eybQCgtEoLppBSj37rtnkqJ0aAkbq1 rjcAoJruTYLiw4fk/Pf1a68OpAnKhqzl =MuWe -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Nick, On 10/25/2011 06:47 PM, Nick Coghlan wrote:
Sure - the symlink option is now there in both PEP and impl. Vinay actually flipped it on by default for platforms where it's reliable, but if you and Barry are both now leaning towards leaving it off by default everywhere, that's equally fine by me. We'll flip it off by default.
This is true. I guess it mostly boils down to making the code in sysconfig less ugly; any external tool should be using sysconfig/site APIs anyway, not doing custom stuff with sys.*_prefix attributes to find site-packages. But a helper function can take care of that ugliness in sysconfig, so I'm ok with your approach to the sys attributes. (Though we still have to pick a name to refer to "sys.venv_prefix if sys.venv_prefix else sys.prefix", it just pushes the naming question onto the helper function :P).
Yep, adding more Windows examples and generally a fuller rationale for the layout to the PEP are next on my list. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nX2YACgkQ8W4rlRKtE2dCWgCePm8irdm52Mz2klfBJBY/DwCt 0bEAn32tOMwjYQmrwZiDEtv2YSkE82Ff =WKEK -----END PGP SIGNATURE-----
Carl Meyer wrote:
Vinay Sajip and I are working on a PEP for making "virtual Python environments" a la virtualenv [1] a built-in feature of Python 3.3.
This will be a very nice feature to have indeed. I've got some questions after reading the draft: 1) There is no mention whatsoever about user site-packages directories from PEP-370. Should they be supported inside venvs? Or should they be considered part of system-wide python? 2) virtualenv keeps using platform-specific layout inside venv. So on POSIX for example this allows to install different python versions and implementations (like cpython and pypy, for example) into the very same venv. OTOH on Windows and in Jython there is only <prefix>\Lib per venv which makes this sharing impossible. Should venvs support such use case? If so, how shebangs should be handled? What layout to endorse? 3) This might be not relevant to this PEP but I wonder how would implementing this proposal affect other implementations like Jython, PyPy and IronPython. Will they be able to implement this functionality the same way? -- Konstantin Zemlyak
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks Konstantin, great questions. On 10/26/2011 07:36 AM, Konstantin Zemlyak wrote:
Currently virtualenv considers user-site-packages part of the global environment; they are not used if you supply --no-site-packages, and are available otherwise. I think this is the right approach: an isolated venv should really be isolated, and it would be odd for venv to provide a mode where user-site-packages is available but global-site-packages are not, since PEP 370 itself has no such provision. I'll add an explicit section on this to the PEP, and verify that the reference implementation handles it correctly. I suppose a case might be made for a mode that makes global-site-packages available but user-site-packages not? My inclination is that this is not necessary, but I will add it as an open question in the PEP, pending further feedback.
I don't think venv should go out of its way to support the case of multiple interpreters in a single venv to any greater degree than Python supports multiple interpreters under a single system prefix. In other words, I'm perfectly happy to let this behavior vary by platform, in the same way it already varies at the system level, and not consider it a supported use case. Virtualenv makes no effort to support sharing an env among multiple interpreters, and this has not been a requested feature. Note that "sharing a venv" in the POSIX layout has very few advantages over having a separate venv per interpreter/version, and some significant disadvantages. You don't gain any disk space savings, and you lose a great deal of explicitness, particularly in that installed scripts will generally have a shebang line pointing to whichever interpreter you happened to install them with, possibly overwriting the same script previously installed by a different interpreter. If people on POSIX systems want to do this, that's fine, but I don't think it should be documented or encouraged.
Good question. Virtualenv already supports PyPy and I'm fairly confident PyPy will not have trouble with this PEP. I don't know about IronPython or Jython - as long as they follow the same algorithm for finding sys.prefix based on sys.executable, it should work the same. I will add an "open question" on this topic to the PEP, and hopefully we can get some feedback from the right people when we take this to python-dev. Thanks for the review! Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oR+QACgkQ8W4rlRKtE2ddegCgxfLsZNtNFsFNxA/bFkBdgBVQ DyYAoJ9Grvu7d4g+EOBKhyHE0l8qwq0x =dfqF -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/24/2011 12:21 PM, Carl Meyer wrote:
I'd like to flip this default and make isolation be the default, with a flag to enable the system site-packages. The non-isolated default I borrowed from virtualenv without a great deal of thought, but we have had requests to flip virtualenv's default, and it seems that many current active virtualenv users (100% of those on the mailing list who have replied) are in favor [1]. It's looking likely that we will make that change in virtualenv; I think venv should do the same. The primary use case for venv is isolation, so why not make it the default? Carl [1] http://groups.google.com/group/python-virtualenv/browse_frm/thread/57e11efac... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oSiYACgkQ8W4rlRKtE2cR6QCeN0Hkuqgjr9kq1HcBp/hl8i26 bYoAn3RorDvJukjcm8OomdjaThUfzXbo =RGNB -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks again to everyone who offered feedback, questions, and suggestions on the draft PEP. I've made substantial revisions and additions to the PEP based on discussion here. A couple changes to call out specifically that I haven't already mentioned: * I've attempted to add more clarity and detail to the discussion of ``sys.prefix`` vs ``sys.site_prefix``, based on Jim's questions. * After discussion with Vinay, we're sticking with ``sys.site_prefix`` naming for now in the PEP and reference implementation. I've added an "open questions" section summarizing the discussion here, the concerns Nick raised about that name, and both the ``sys.venv_prefix`` and ``sys.local_prefix`` alternative suggestions. We'll see what additional feedback there is from python-dev on that question. Sending the draft on to the PEP editors and python-dev now, as outlined in PEP 1. Thanks! Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6qzuIACgkQ8W4rlRKtE2eS2ACfQmWpO4lsxYvZdhJaDqCC3X9j GkYAn0P7dtgk0GbsAXA/36mvTONGa127 =3Sck -----END PGP SIGNATURE-----
On Tue, Oct 25, 2011 at 4:21 AM, Carl Meyer <carl@oddbird.net> wrote:
The repeated references to copying binaries and Python files throughout the PEP is annoying, and needs to be justified. Python 3.2+ supports symlinking on Windows Vista and above as well as on *nix systems, so there needs to be a short section somewhere explaining why symlinks are not an adequate lightweight solution (pointing out the fact that creating symlinks on Windows often requires administrator privileges would be sufficient justification for me). (Obviously, 3rd party virtual environment solutions generally *do* have to copy things around, since Python 2.x doesn't expose symlink support on Windows at all)
It can also take advantage of the native symlink support to minimise copying.
Currently, the PEP uses a mish-mash of 'env', 'venv', 'pyvenv' and 'pythonv' and 'site' to refer to different aspects of the proposed feature. I suggest standardising on 'venv' wherever the Python relationship is implied, and 'pyvenv' wherever the Python relationship needs to be made explicit. So I think the name of the configuration file should be "pyvenv.cfg" (tangent: 'setup' lost its Python connection when it went from 'setup.py' to 'setup.cfg'. I wish the latter has been called 'pysetup.cfg' instead)
'site' is *way* too overloaded already, let's not make it worse. I suggest "sys.venv_prefix".
The builtin virtual environment mechanism should be specified to symlink things by default, and only copy things if the user specifically requests it. System administrators rightly fear the proliferation of multiple copies of binaries, since it can cause major hassles when it comes time to install security updates.
"site" is ambiguous here - rather than abbreviating, I suggest making the option "include-system-site-packages".
As noted above, those should be symlinks rather than copies, with copying behaviour explicitly requested via a command line option. Also, why "Scripts" rather than "bin" on Windows? The Python binary isn't a script. I'm actually not seeing the rationale for the obfuscated FHS inspired layout in the first place - why not dump the binaries adjacent to the config file, with a simple "site-packages" directory immediately below that? If there are reasons for a more complex default layout, they need to be articulated in the PEP. If the problem is wanting to allow cross platform computation of things like the site-packages directory location and other paths, then the answer to that seems to lie in better helper methods (whether in sysconfig, site, venv or elsewhere) rather than Linux specific layouts inside language level virtual environments.
Yikes, double negatives in APIs are bad news (especially when the corresponding config file option is expressed positively) I suggest this parameter should be declared as "system_site_packages=True".
In line with my above comments, I think there should be a third parameter here declared as "use_symlinks=True". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Oct 25, 2011 at 8:31 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
It's justification for supporting it, not necessarily for doing it implicitly (although Windows in general is far more tolerant of bundling than the various *nix platforms). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Oct 25, 2011 at 6:11 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Yeah, I realised I don't actually mind if things get copied around on Windows - it's the POSIX systems where implicit copying would bother me, and that goes to the heart of a longstanding difference in packaging philosophy between the two platforms :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10/25/2011 4:28 AM, Nick Coghlan wrote:
I have a different issue with copying. I have a new Win7 system with a solid state disk for programs (that usually sit unchanged months or years at a time). It is nice and fast, but has a finite write-cycle lifetime. So unnecessary copies are not nice. I presume venv could be told to copy onto the data disk, but then everything runs slower. -- Terry Jan Reedy
On Oct 26, 11:26 am, Terry Reedy <tjre...@udel.edu> wrote:
As you said, though, you're using it for primarily static programs. If you're concerned with its lifetime, would you really use it for development? The onus that SSD afficiendos place on others to not use drives as writeable data stores seems absolutely crazy to me.
On 10/25/2011 11:24 PM, alex23 wrote:
I don't. I put a (static) .pth file in site-packages that points to my development directory on the hard disk, where I happily edit and test sometimes several times an hour.
The onus that SSD afficiendos place on others to not use drives as writeable data stores seems absolutely crazy to me.
I said a) I would prefer to not have an unnecessary duplication but b) if a copy is to be made, I would like to have a choice of which drive to use. That does not strike me as 'absolutely crazy'. -- Terry Jan Reedy
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Terry, On 10/25/2011 10:50 PM, Terry Reedy wrote:
As things stand with the PEP, you can create virtualenvs wherever you like, and you can optionally use symlinks if you prefer. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nmegACgkQ8W4rlRKtE2dMwQCfevz5GgWK0nl3O4vqegsk902/ JTwAn2yu/MAb9a8nAYk0SJMfO/b32Yih =C3jN -----END PGP SIGNATURE-----
On 26 October 2011 14:24, alex23 <wuwei23@gmail.com> wrote:
As such an afficionado, I agree that trying not to use it as a writeable data store seems crazy. All current SSDs have a sufficient number of write-erase cycles that you would need to be writing to the entire drive non-stop for years before it became an issue. If you're worried, get more RAM and configure a RAM disk as your temporary store. If nothing else, it speeds up compiles significantly ... Tim Delaney
On Tue, 25 Oct 2011 21:26:07 -0400 Terry Reedy <tjreedy@udel.edu> wrote:
Uh, please do not throw around arguments like that without backing them up with concrete numbers. In practice, your SSD will have amply enough write-cycles for venv to work fine. And if it doesn't, you've simply been swindled. Regards Antoine.
----- Original Message -----
I agree that symlinking should be done wherever possible, but let's remember that it's not just a Python issue: Windows XP does not support true symlinks, but only "junctions" aka "reparse points". Of course, that's no reason not to use true symlinks where they *are* available.
On Mon, Oct 24, 2011 at 20:16, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
On 3.2+, symlinks on Windows are possible but only in what I suspect is a pretty rare version and environment combination. As said earlier, they work on Vista and beyond, but they require that the process has been elevated in order to obtain the symlink privilege. For example, even as an administrator account on Windows 7, you need to explicitly open up cmd with "Run as Administrator" to properly symlink (through Python or otherwise). If we can't execute the symlink, we raise OSError. We might as well try symlinks then fallback on OSError.
On Tue, Oct 25, 2011 at 11:42 AM, Brian Curtin <brian.curtin@gmail.com> wrote:
I'd prefer some kind of warning if the symlink fails on a POSIX system. On Windows, bundling is such an accepted practice that even copying by default wouldn't bother me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Vinay Sajip wrote:
Windows XP does not support true symlinks, but only "junctions" aka "reparse points".
Out of curiosity, how far do these fall short of being true symlinks? The points I'm aware of are: * They only work within a volume * The GUI doesn't know about them, so it's easy to mistake a link to a folder for an independent copy of it and accidentally trash the original Are there any others? -- Greg
On 25 October 2011 16:16, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Junctions are only available for directories, and they're more akin in practice to hardlinks than symlinks (although different to both). Junctions can quite happily work cross-volume. It's hardlinks that only work within a volume (and you can't make a hardlink to a directory). Junctions are transparent within a network share i.e. if you a share that contains a junction, the junction will be traversed correctly. Symlinks do not work as you would expect when accessed via a network share. On Win7 and Vista, deleting a junction only deletes the junction. On XP deleting a junction deletes the contents as well unless Link Shell Extension <http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html> is installed. Tim Delaney
On 25 October 2011 06:16, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
They are very uncommon on Windows in general, so people find them confusing when they encounter them (for example, as a consequence of the GUI issue mentioned above - I know that's happened to me). Is there any reason that hard links can't be used on Windows? (Still not cross-volume, still relatively rarely used, but a bit better known than symlinks). BTW, regardless of whether symlinks, hardlinks or copies end up being used, I'm +1 on the general proposal. Paul.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Nick, Thanks for the feedback, replies below. On 10/24/2011 04:11 PM, Nick Coghlan wrote:
Do you mean pointing out why symlinks are not in themselves an adequate solution for virtual Python environments, or why they aren't used in this implementation? I've updated the introductory paragraph you quoted to provide a bit more detail on the first question. I think the answer to the latter question is just that we were trying to keep things simple and consistent, and make it work the same way on as wide a variety of platforms as possible (e.g. Windows XP). Also, in earlier discussions on distutils-sig some people considered it a _feature_ to have the virtual environment's Python binary copied, making the virtual environment more isolated from system changes. Obviously, this is an area where programmers and sysadmins often don't see eye to eye ;) The technique in this PEP works just as well with a symlinked binary, though, and I don't see much reason not to provide a symlink option. Whether it is on by default where supported is something that may need more discussion (I don't personally have a strong opinion either way). The reason virtualenv copies rather than symlinks the binary has nothing to do with lack of symlink support in Python 2, it's because getpath.c dereferences a symlinked binary in finding sys.prefix, so virtualenv's isolation technique simply doesn't work at all with a symlinked binary. Our version with the config file and Vinay's changes in getpath.c does work with a symlinked binary.
I don't think this is a significant enough difference to warrant mention here. Existing virtualenv on Python 2 already can and does use symlinks (for the bits of the stdlib it needs) on platforms that have os.symlink. IOW, the details and supported platforms differ, but on both Python 2 and 3 the best you can do is try to use os.symlink where available, and be prepared to fall back to a copy.
Good point. We were already attempting to standardize just as you suggest, but hadn't renamed "env.cfg", and I missed one remaining instance of "pythonv" in the text (it's also used for legacy reasons in the reference implementation bitbucket repo name, but that doesn't seem worth changing).
This makes sense to me. I've updated the PEP accordingly and created an issue to remind me to update the reference implementation as well.
'site' is *way* too overloaded already, let's not make it worse. I suggest "sys.venv_prefix".
My original thinking here was that sys.site_prefix is an attribute that should always exist, and always point to "where stuff should be installed to site-packages", whether or not you are in a venv (if you are not, it would have the same value as sys.prefix). It's a little odd to use an attribute named "sys.venv_prefix" in that way, even if your code doesn't know or care whether its actually in a venv (and in general we should be encouraging code that doesn't know or care). (The attribute doesn't currently always-exist in the reference implementation, but I'd like to change that). I agree that "site" is overloaded, though. Any ideas for a name that doesn't further overload that term, but still communicates "this attribute is a standard part of Python that always has the same meaning whether or not you are currently in a venv"?
To be clear, "things" here refers only to the Python binary itself. The only other things that might be installed in a new environment are scripts (e.g. pysetup3), and those must be created anew, neither symlinked nor copied, as their shebang line needs to point to the venv's Python (or more complicated chicanery with .exe wrappers on Windows, unless we get PEP 397 in time).
I think both options should be allowed, and I don't have a strong feeling about the default.
Ok - updated the draft PEP, will update reference implementation.
Also, why "Scripts" rather than "bin" on Windows? The Python binary isn't a script.
No, but in real-world usage, scripts from installed packages in the virtualenv will be installed there. Putting the Python binary in the same location as the destination for installed scripts is a pretty important convenience, as it means you only need to add a single directory to the beginning of your shell path to effectively "activate" the venv.
The historical reason is that it emulates the layout found under sys.prefix in a regular Python installation (note that it's not actually Linux-specific, in virtualenv it matches the appropriate platform; i.e. on Windows it's "Lib\" rather than "lib\pythonX.X"). This was necessary for virtualenv because it couldn't make changes in distutils/sysconfig). I think there may be good reason to continue to follow this approach, simply because it makes the necessary changes to distutils/sysconfig less invasive, reducing the need for special-casing of the venv case. But I do need to look into this a bit more and update the PEP with further rationale in any case. Regardless, I would not be in favor of dumping binaries directly next to pyvenv.cfg. It feels cleaner to keep scripts and binaries in a directory specifically named and intended for that purpose, which can be added to the shell PATH. I also think there is some value, all else being roughly equal, in maintaining consistency with virtualenv's layout. This is not an overriding concern, but it will make a big difference in how much existing code that deals with virtual environments has to change.
Fair enough, updated in draft PEP.
Thanks again for the review! The updated draft is available on Bitbucket [1], and the open issues for the reference implementation (which should reflect outstanding differences from the draft PEP) are as well [2]. Carl [1] https://bitbucket.org/carljm/pythonv-pep/src [2] https://bitbucket.org/vinay.sajip/pythonv/issues?status=new&status=open -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6m8+gACgkQ8W4rlRKtE2f1EgCfZNLBXSI08UQdLCRQMYwxwAp3 ByoAn3cVYvQXWMc1xkoO6mMSmNBQbEAD =FzBA -----END PGP SIGNATURE-----
On 25 October 2011 18:37, Carl Meyer <carl@oddbird.net> wrote:
One thought - on "per user" Windows installations, or uninstalled Python builds on Windows, IIRC python.exe requires pythonXY.dll to be alongside the EXE. You should probably consider copying/linking that if it's not installed systemwide. It's not entirely obvious to me how you'd detect this, though - maybe take the simple approach of copying python.exe, and also copying pythonXY.dll if it's in the same directory otherwise assume it's a system install. There's also pythonw.exe on Windows, of course... (I'm assuming that w9xpopen.exe is probably not needed - I'm not even sure there's any remaining need to ship it with Python in the first place). Paul.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Paul, On 10/25/2011 12:11 PM, Paul Moore wrote:
Actually, the reference implementation does symlink/copy all DLLs from the same directory as the python binary or a DLLs/ subdirectory, and also any executable beginning with "python", which would include pythonw.exe. We should certainly update the PEP with more detail about how things look different on Windows, currently it's pretty much describing the POSIX behavior (my fault; Vinay's done all the Windows work and I'm not as familiar with it). Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nAjQACgkQ8W4rlRKtE2dF4gCfaBpwyQArhArI6ybo9c6Qbjsz mekAn3/LnNHnt3U79ZA6GIyZXxxE4Ja2 =whkG -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Oct 25, 2011, at 11:37 AM, Carl Meyer wrote:
I definitely think folks will want copy and symlink options, for exactly the reason you and Nick point out. As for what the default should be, I think maximal isolation probably wins, and thus copying should be the default. I wouldn't scream if it were the other way 'round though.
That makes sense to me. It would be nice to have the location of your site-prefix always available for calculations regardless of your "virtualness". Note that in all likelihood, Debian will have to modify that so that it points to dist-packages instead of site-packages. Will there be an easy hook/config file or some such that we could use to make that change?
sys.addons_directory, sys.plugins_directory, sys.external_packages or some such?
+1
All of that makes sense to me, so adding these rationales to the PEP would be useful. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJOpx1XAAoJEBJutWOnSwa/I2YP/jXyJCu5uIVp20H+SjlqlnmB y5tbzgSTPi3Yf1In1+bs1bPdNhqlZYi1WfmGRPA7QfXWK4oeSpPXTjs/bAaTYmPE GjWDvQKue1hNv8qdFbo0IyI3u+2Zq0KB3sC+ACxB8WZ/Xc9wt4q47zMOTods4/Zj ct5nvsib6xF/ZGk2BJopEQwFHhraxuNkrhSsgbUUsQJ3XEmVtLQZ4Dyr5gxRDmOF /oShU556C3JjyPsXkNmURjQgXsbsWN/7j815yVpsLWNm5fjCPjqJAjY64RBFe9l/ EJqutLvqfQJJ1ie/R8xBBLF1LWNzzOeWgcgenbrdlgXa7xuGQE8psb1CSxdEUgec 9CUqsZbXDhQDtgdIcqsemhtGSIrl8UtCyrIDj7HUqK6ibOLTCa83AztpyupXpcki ynTwJLdSYRc+8IH6kLQ5DpqYCYC+14jEoMrDSSRGB/EyAE1P0l3DE710JtYHV12R 6e0Ciw8qjuhUTnmAr6H5DnT8UaWXENBDkDbel6ABHxEKalavAnr3kzjUTLcnTwMJ QfrbcVDBRnGV1swelLZ3oXitgofCCaz7Ca4kwqHikUQ5R/iPNl5AJyyJenFHsPrc J0jfSQEqPP3FOO27atPlVy/9lE45Ac0D7UcMq5IrfruLd4Rw7P50sPIxa+anAfKK GEuw81sCOyyJ4PU0cfxw =KObT -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Barry, Thanks for the review and feedback. On 10/25/2011 02:34 PM, Barry Warsaw wrote:
As it currently stands, this is really just the/a _prefix_ under which site-packages is found (parallel to how sys.prefix is currently used). Which means "site-packages" vs "dist-packages" is still determined in site.py, which tacks platform-specific paths onto the end of the prefix in order to find the actual site-packages directory. So in the Debian case, you'd still have to modify your distributed site.py (and take a bit of care, assisted by the tests, that you don't break venv.py in the process); I don't think this PEP really changes that much. (Though the fact that not breaking virtual environments when you make distro-specific modifications to site.py becomes your responsibility as the distro Python maintainer rather than mine as virtualenv maintainer is, in my humble opinion, one of the greatest advantages of this PEP <wink>) Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nYLEACgkQ8W4rlRKtE2ftGgCfQr0m9WM5BtDxFZfqepbxqqxn +kIAniTBdvxWh/Bsg4rOEdGMYnIIpKmD =CMax -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Oct 25, 2011, at 07:21 PM, Carl Meyer wrote:
Yes, fair enough! :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJOp2b5AAoJEBJutWOnSwa/irAQAMN+ZtkWAVRdcs9XosYyWsaT gs/BiAMJ2pob0WZ+U0/8BDM5/uz7UYoPqzmn6EzDKKA+ezhJ8UNYwWTkQoUpuQK3 OqTAA06qJ3oVR+BYoC8kzriTWNxuJ6jCdJbRrK8PXTz3FBdhw720mHO1A5k3ZmEA 2HebsbO7X11iU5JyNV4+K98DzUna/RnLvT3RUm8NPjUpBIS5QHhnVpmFEHrEy1F0 QnyZXbY7UqpUxXtBY6Q1sHoERRmZ2EZOeHwzBMC0iUcig3Pmps+taxHqXxhPh72F cNSt76K/ceHlNQrB8pBKQnP2FUn5KrZ8ut+LNHQmP6LYSZ/2r0kExbbqzF7xIxce ACfIl1H/ezzA7KK29WlKMgqMU6K/OL2pcdtZry35wKCA1RIghpds44iEV26yqWPK EY428lmWrkPZ2YGsfGyrm5E5vOzWEmcp7ze9FY34vgiphMzAkl1opCqKRffSc+ci dzmmjAOot+J6jVX0zAPqwI03Uys8/HJZwY8kK5VyKJon5EVI5FI9C/gqBDwR/2vP QZzdnlswwgGSz9xclzOhPW66lCZ5duNC115evNBQ030zoQRnGNICc+SNRB2GYY2v gzutj9KT2sB+HuvQagQfk8zuLs5pCSd9jLxXb5JPxLb8NCJsxHG8LrZfm+HiVD58 QOUkGdfxEDv76AD4f6mu =psL3 -----END PGP SIGNATURE-----
On Wed, Oct 26, 2011 at 3:37 AM, Carl Meyer <carl@oddbird.net> wrote:
I've been persauded that copying by default is reasonable - so long as the option to use symlinks is there, sysadmins can put a policy for their organisation in place that requires its use.
I'd actually prefer that we use the explicit "sys.prefix" and "sys.venv_prefix" naming (with the latter set to None when not in a virtual env) and possibly offer a convenience API somewhere that hides the "sys.prefix if sys.venv_prefix is None else sys.venv_prefix" dance. There's already a site.getsitepackage() API that gets the full list of site package directories (it's not a given that there's only one).
As Barry said, the rationale sounds reasonable, but the PEP needs to explain it in terms those of us not fully versed in the details of cross-platform virtual environments can understand (examples of both *nix and Windows layouts with a simple example package installed would probably be useful on that front) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Oct 26, 2011 at 11:51 AM, Barry Warsaw <barry@python.org> wrote:
Yeah, having venv_prefix == prefix in the "not in a virtual env case" is fine by me as well. I think Carl's right that it reads a little oddly sometimes, but it's a better option than: - further overloading "site" (when more than site-package may be located using the venv prefix) - requiring people to fall back to sys.prefix explicitly The "am I in a virtual env?" check can then just be "if sys.prefix == sys.venv_prefix". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/25/2011 08:05 PM, Nick Coghlan wrote:
What about "sys.local_prefix"? Doesn't overload site, but also doesn't imply that it always points to a venv. I think "local" carries the right connotation here - this is the prefix for the user's local environment (as possibly opposed to the "global" system environment). Seems like this might make code using the attribute unconditionally read a bit less oddly?
The "am I in a virtual env?" check can then just be "if sys.prefix == sys.venv_prefix".
Right; though in general the goal is that by simply switching to use the new prefix in the right places in sysconfig, such an explicit "am in a venv" check should never be necessary, even in the stdlib. I think the reference implementation largely achieves this goal, I'd have to check again to see if that's entirely true. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nbPMACgkQ8W4rlRKtE2dbrQCg3iwUkXZUBHerzyBFq+jOWS91 pgkAn2hYy7pREwdFBQQayR2OLP9x62e9 =tx9A -----END PGP SIGNATURE-----
On Wed, Oct 26, 2011 at 12:14 PM, Carl Meyer <carl@oddbird.net> wrote:
I think explaining to people "sys.venv_prefix is still valid when you're not in a virtual env, it just points to the same place as sys.prefix" is easier than explaining a *new* name with "<whatever name> points to the virtual env directory when you're in a virtual environment and sys.prefix otherwise". So while I agree your concern is valid, I think just living with the quirkiness is a reasonable approach. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan@...> writes:
I thought Carl was saying that we use "local_prefix" as opposed to "venv_prefix", because the "local_" prefix seems logical and doesn't override "site". In a venv, local_prefix/local_exec_prefix point to the venv, otherwise they have the same values as prefix/exec_prefix. The "venv_XXX" does grate a bit, especially as we're trying to achieve a don't-know-or-care-if- I'm-in-a-venv as much as possible. And thinking more about overriding "site" - what is a "site" anyway? It seems to be a combination of Python version, standard library and a specific set of packages (comprising in the general case include files, shared libs and extension modules) - if I installed 3 different versions of Python on my system, I would have three different sites. By that definition, a venv is really a site, albeit a pared-down one which references shared code and library where possible. So site_prefix/site_exec_prefix don't seem unreasonable to me, or failing that, local_prefix/local_exec_prefix would be preferable to venv_prefix/venv_exec_prefix. Regards, Vinay Sajip
On Oct 26, 2011, at 09:02 AM, Vinay Sajip wrote:
I agree in general with these observations. What we're currently calling "site" feels more like "system" to me, although we probably can't change that until Python 4000. One other reason I like "local" over "venv" is that "venv" isn't actually a word. While it'll probably make it easier to google, it looks fairly jarring to me. In general I try to avoid made up words and abbreviations because they are harder for non-native English speakers to use (so I've been told by some non-native English speakers). -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Nick, On 10/25/2011 08:26 PM, Nick Coghlan wrote:
FWIW, I agree with Vinay and Barry here. It's not something I care deeply about (since I think direct use of these sys attributes should be discouraged outside the stdlib anyway), but I don't agree that a misnamed attribute is easier to explain than a new name. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oTYMACgkQ8W4rlRKtE2dXbgCggLUDFr1cqRrOTlvPH9PoC0y5 Y7QAoMyQtRh22P2ly7TmrS3/SlQncFFd =swWE -----END PGP SIGNATURE-----
On Tue, Oct 25, 2011 at 9:15 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Nick Coghlan <ncoghlan@...> writes:
But why is that better than a site.venv_prefix which points to a venv if you're in one, and == sys.prefix if you're not?
Is there a reason we can't just make sys.* be the virtual environment's version, and sys.base.* be the site-wide version? You could still find sys.base.*, and you could still check whether sys.prefix == sys.base.prefix, but programs that don't get updated will use the sandboxed virtual version by default. -jJ
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jim, On 10/26/2011 10:04 AM, Jim Jewett wrote:
Yes, this is a reasonable alternative that Vinay and I discussed at some length earlier in development. There are reasonable arguments to be made in both directions - you can read my summary of those arguments in the "Open Questions" section of the PEP. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oPh4ACgkQ8W4rlRKtE2ftFwCgmmTHdduy6C4f8fihMs2Q0/Nw E18AoMAExZGageDgI+1BzikBsrzQCYd+ =+bJn -----END PGP SIGNATURE-----
On Wed, Oct 26, 2011 at 1:06 PM, Carl Meyer <carl@oddbird.net> wrote:
On 10/26/2011 10:04 AM, Jim Jewett wrote:
Even after re-reading, I'm not sure I fully understand. As best I can tell, the question is whether sys.prefix should point to the base system (because of header files?) or the virtual sandbox (because of user site packages?) Will header files and such be explicitly hidden from applications running in the virtual environment? It sounded like the answer was "depends on the flag value". But it seems to me that: (a) If the base system's data (such as headers) are explicitly hidden, then there isn't much good reason to then say "oh, by the way, the secret is hidden over here". (b) If they are not explicitly hidden, then they (or more-correct overrides) should also be available from the virtual environment (whether by copying or by symlink, or even by symlink to the whole directory). So I'm still not seeing any situation where a normal program should care about the base value, but I am seeing plenty of situations where it might erroneously ask for that value and test (or install) something outside the virtual environment as a result. -jJ
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/26/2011 12:28 PM, Jim Jewett wrote:
The standard library is also located always in the base system only.
This dichotomy assumes an even more comprehensive form of isolation (basically that which is provided already by PYTHONHOME). That level of isolation is problematic precisely because it requires either copying or symlinking the entire standard library into every environment (essentially duplicating the base Python installation for every environment, perhaps using symlinks to gain some efficiency). This is an issue because: 1) Not all supported platforms have good support for symlinks, and copying the entire standard library is quite heavyweight if you'll be creating a lot of environments. 2) site-packages is contained within the standard library directory, and you presumably want to override site-packages, so you can't just make a single "lib" symlink to get the stdlib, you have to individually symlink every single module. If you aren't bothered by these caveats, then you can just use PYTHONHOME today, and you have no need for this PEP. The goal of this PEP is to provide site-packages isolation without standard-library isolation (and its attendant downsides), by allowing knowledge of the base system environment (enough to use its standard library) even when Python is running within the virtual environment. The Google code search linked in the PEP demonstrates that there is code out there using sys.prefix to find the standard library (for instance, to exclude it from tracing). This code will no longer function if sys.prefix no longer points to the location where the standard library is actually found. It's possible that there is more code out there using sys.prefix to find site-packages. However, the documented definition of sys.prefix never mentions site-packages, whereas it does mention the standard library and header files. So if we have to choose which code to break, I see a strong argument for adhering to the documented definition rather than breaking it. (I could fairly easily be convinced, however, that practicality beats purity in this case, especially since changing sys.prefix errs in the direction of greater isolation.) (It's worth noting that really no code outside the stdlib should be using any of these sys prefix attributes directly, it should instead use the appropriate APIs in site/sysconfig/distutils).
The case you are missing is "any code which cares where the standard library or header files are located and is using sys.prefix to figure this out," because it is a design goal of this PEP to not have to provide indirection for those things in the virtual environment layout. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oV9UACgkQ8W4rlRKtE2eybQCgtEoLppBSj37rtnkqJ0aAkbq1 rjcAoJruTYLiw4fk/Pf1a68OpAnKhqzl =MuWe -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Nick, On 10/25/2011 06:47 PM, Nick Coghlan wrote:
Sure - the symlink option is now there in both PEP and impl. Vinay actually flipped it on by default for platforms where it's reliable, but if you and Barry are both now leaning towards leaving it off by default everywhere, that's equally fine by me. We'll flip it off by default.
This is true. I guess it mostly boils down to making the code in sysconfig less ugly; any external tool should be using sysconfig/site APIs anyway, not doing custom stuff with sys.*_prefix attributes to find site-packages. But a helper function can take care of that ugliness in sysconfig, so I'm ok with your approach to the sys attributes. (Though we still have to pick a name to refer to "sys.venv_prefix if sys.venv_prefix else sys.prefix", it just pushes the naming question onto the helper function :P).
Yep, adding more Windows examples and generally a fuller rationale for the layout to the PEP are next on my list. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6nX2YACgkQ8W4rlRKtE2dCWgCePm8irdm52Mz2klfBJBY/DwCt 0bEAn32tOMwjYQmrwZiDEtv2YSkE82Ff =WKEK -----END PGP SIGNATURE-----
Carl Meyer wrote:
Vinay Sajip and I are working on a PEP for making "virtual Python environments" a la virtualenv [1] a built-in feature of Python 3.3.
This will be a very nice feature to have indeed. I've got some questions after reading the draft: 1) There is no mention whatsoever about user site-packages directories from PEP-370. Should they be supported inside venvs? Or should they be considered part of system-wide python? 2) virtualenv keeps using platform-specific layout inside venv. So on POSIX for example this allows to install different python versions and implementations (like cpython and pypy, for example) into the very same venv. OTOH on Windows and in Jython there is only <prefix>\Lib per venv which makes this sharing impossible. Should venvs support such use case? If so, how shebangs should be handled? What layout to endorse? 3) This might be not relevant to this PEP but I wonder how would implementing this proposal affect other implementations like Jython, PyPy and IronPython. Will they be able to implement this functionality the same way? -- Konstantin Zemlyak
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks Konstantin, great questions. On 10/26/2011 07:36 AM, Konstantin Zemlyak wrote:
Currently virtualenv considers user-site-packages part of the global environment; they are not used if you supply --no-site-packages, and are available otherwise. I think this is the right approach: an isolated venv should really be isolated, and it would be odd for venv to provide a mode where user-site-packages is available but global-site-packages are not, since PEP 370 itself has no such provision. I'll add an explicit section on this to the PEP, and verify that the reference implementation handles it correctly. I suppose a case might be made for a mode that makes global-site-packages available but user-site-packages not? My inclination is that this is not necessary, but I will add it as an open question in the PEP, pending further feedback.
I don't think venv should go out of its way to support the case of multiple interpreters in a single venv to any greater degree than Python supports multiple interpreters under a single system prefix. In other words, I'm perfectly happy to let this behavior vary by platform, in the same way it already varies at the system level, and not consider it a supported use case. Virtualenv makes no effort to support sharing an env among multiple interpreters, and this has not been a requested feature. Note that "sharing a venv" in the POSIX layout has very few advantages over having a separate venv per interpreter/version, and some significant disadvantages. You don't gain any disk space savings, and you lose a great deal of explicitness, particularly in that installed scripts will generally have a shebang line pointing to whichever interpreter you happened to install them with, possibly overwriting the same script previously installed by a different interpreter. If people on POSIX systems want to do this, that's fine, but I don't think it should be documented or encouraged.
Good question. Virtualenv already supports PyPy and I'm fairly confident PyPy will not have trouble with this PEP. I don't know about IronPython or Jython - as long as they follow the same algorithm for finding sys.prefix based on sys.executable, it should work the same. I will add an "open question" on this topic to the PEP, and hopefully we can get some feedback from the right people when we take this to python-dev. Thanks for the review! Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oR+QACgkQ8W4rlRKtE2ddegCgxfLsZNtNFsFNxA/bFkBdgBVQ DyYAoJ9Grvu7d4g+EOBKhyHE0l8qwq0x =dfqF -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/24/2011 12:21 PM, Carl Meyer wrote:
I'd like to flip this default and make isolation be the default, with a flag to enable the system site-packages. The non-isolated default I borrowed from virtualenv without a great deal of thought, but we have had requests to flip virtualenv's default, and it seems that many current active virtualenv users (100% of those on the mailing list who have replied) are in favor [1]. It's looking likely that we will make that change in virtualenv; I think venv should do the same. The primary use case for venv is isolation, so why not make it the default? Carl [1] http://groups.google.com/group/python-virtualenv/browse_frm/thread/57e11efac... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oSiYACgkQ8W4rlRKtE2cR6QCeN0Hkuqgjr9kq1HcBp/hl8i26 bYoAn3RorDvJukjcm8OomdjaThUfzXbo =RGNB -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks again to everyone who offered feedback, questions, and suggestions on the draft PEP. I've made substantial revisions and additions to the PEP based on discussion here. A couple changes to call out specifically that I haven't already mentioned: * I've attempted to add more clarity and detail to the discussion of ``sys.prefix`` vs ``sys.site_prefix``, based on Jim's questions. * After discussion with Vinay, we're sticking with ``sys.site_prefix`` naming for now in the PEP and reference implementation. I've added an "open questions" section summarizing the discussion here, the concerns Nick raised about that name, and both the ``sys.venv_prefix`` and ``sys.local_prefix`` alternative suggestions. We'll see what additional feedback there is from python-dev on that question. Sending the draft on to the PEP editors and python-dev now, as outlined in PEP 1. Thanks! Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6qzuIACgkQ8W4rlRKtE2eS2ACfQmWpO4lsxYvZdhJaDqCC3X9j GkYAn0P7dtgk0GbsAXA/36mvTONGa127 =3Sck -----END PGP SIGNATURE-----
participants (14)
-
alex23 -
Antoine Pitrou -
Barry Warsaw -
Brian Curtin -
Carl Meyer -
Ethan Furman -
Greg Ewing -
Jim Jewett -
Konstantin Zemlyak -
Nick Coghlan -
Paul Moore -
Terry Reedy -
Tim Delaney -
Vinay Sajip