Re: [Python-Dev] PEP 405 (built-in virtualenv) status
On 03/15/2012 03:02 PM, Lindberg, Van wrote:
FYI, the location of the tcl/tk libraries does not appear to be set in the virtualenv, even if tkinter is installed and working in the main Python installation. As a result, tk-based apps will not run from a virtualenv.
Thanks for the report! I've added this to the list of open issues in the PEP and I'll look into it. Carl
Hi Carl. I'm very interested in this work. At CCP we work heavily with virtual environments. Except that we don't use virtualenv because it is just a pain in the neck. We like to be able to run virtual python environments of various types as they arrive checked out of source control repositories, without actually "installing" anything. For some background, please see: http://blog.ccpgames.com/kristjan/2010/10/09/using-an-isolated-python-exe/. It's a rather quick read, actually. The main issue for us is: How to prevent your local python.exe from reading environment variables and running some global site.py? There are a number of points raised in the above blog, please take a look at the "Musings" at the end. Best regards, Kristján -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com@python.org [mailto:python-dev-bounces+kristjan=ccpgames.com@python.org] On Behalf Of Carl Meyer Sent: 15. mars 2012 22:12 To: python-dev Subject: Re: [Python-Dev] PEP 405 (built-in virtualenv) status On 03/15/2012 03:02 PM, Lindberg, Van wrote:
FYI, the location of the tcl/tk libraries does not appear to be set in the virtualenv, even if tkinter is installed and working in the main Python installation. As a result, tk-based apps will not run from a virtualenv.
Thanks for the report! I've added this to the list of open issues in the PEP and I'll look into it. Carl
On 3/19/2012 2:26 AM, Kristján Valur Jónsson wrote:
Hi Carl. I'm very interested in this work. At CCP we work heavily with virtual environments. Except that we don't use virtualenv because it is just a pain in the neck. We like to be able to run virtual python environments of various types as they arrive checked out of source control repositories, without actually "installing" anything. For some background, please see:http://blog.ccpgames.com/kristjan/2010/10/09/using-an-isolated-python-exe/. It's a rather quick read, actually.
The main issue for us is: How to prevent your local python.exe from reading environment variables and running some global site.py?
There are a number of points raised in the above blog, please take a look at the "Musings" at the end.
Best regards,
Kristján
I found that a very interesting reverse-engineering of what needs to be done to isolate multiple pythons on a machine. I concur that this is a feature that would be good to: 1) at least document the behavior well 2) preferably make an extensible feature, along the lines that Kristján suggests There are likely some bootstrapping issues, but I find the idea that the difference between an embedded Python and an installed Python and a built-but-not-installed Python being conceptually isolated to the python.exe and/or site.py rather than python.dll to be a clever concept; of course, where the code lives is less relevant than the conditions under which it is invoked; I doubt the size of the code is the issue regarding where it lives.
Hello Kristján, On 03/19/2012 03:26 AM, Kristján Valur Jónsson wrote:
Hi Carl. I'm very interested in this work. At CCP we work heavily with virtual environments. Except that we don't use virtualenv because it is just a pain in the neck. We like to be able to run virtual python environments of various types as they arrive checked out of source control repositories, without actually "installing" anything. For some background, please see: http://blog.ccpgames.com/kristjan/2010/10/09/using-an-isolated-python-exe/. It's a rather quick read, actually.
The main issue for us is: How to prevent your local python.exe from reading environment variables and running some global site.py?
There are a number of points raised in the above blog, please take a look at the "Musings" at the end.
Thanks, I read the blog post. I think there are some useful points there; I too find the startup sys.path behavior of Python a bit more complex and magical than I'd prefer (I'm sure it's grown organically over the years to address a variety of different needs and concerns, not all of which I understand or am even aware of). I think there's one important (albeit odd and magical) bit of Python's current behavior that you are missing in your blog post. All of the initial sys.path directories are constructed relative to sys.prefix and sys.exec_prefix, and those values in turn are determined (if PYTHONHOME is not set), by walking up the filesystem tree from the location of the Python binary, looking for the existence of a file at the relative path "lib/pythonX.X/os.py" (or "Lib/os.py" on Windows). Python takes the existence of this file to mean that it's found the standard library, and sets sys.prefix accordingly. Thus, you can achieve reliable full isolation from any installed Python, with no need for environment variables, simply by placing a file (it can even be empty) at that relative location from the location of your Python binary. You will still get some default paths added on sys.path, but they will all be relative to your Python binary and thus presumably under your control; nothing from any other location will be on sys.path. I doubt you will find this solution satisfyingly elegant, but you might nonetheless find it practically useful. The essence of PEP 405 is simply to provide a less magical way to achieve this same result, by locating a "pyvenv.cfg" file next to (or one directory up from) the Python binary. The bulk of the work in PEP 405 is aimed towards a rather different goal from yours - to be able to share an installed Python's copy of the standard library among a number of virtual environments. This is the purpose of the "home" key in pyvenv.cfg and the added sys.base_prefix (which point to the Python installation whose standard library will be used). I think this serves a valuable and common use case, but I wonder if your use case couldn't also be served with a minor tweak to PEP 405. Currently it ignores a pyvenv.cfg file with no "home" key; instead, it could set sys.prefix and sys.base_prefix both to the location of that pyvenv.cfg. For most purposes, this would result in a broken Python (no standard library), but it might help you? Beyond that possible tweak, while I certainly wouldn't oppose any effort to clean up / document / make-optional Python's startup sys.path-setting behavior, I think it's mostly orthogonal to PEP 405, and I don't think it would be helpful to expand the scope of PEP 405 to include that effort. Carl
Carl Meyer wrote:
The bulk of the work in PEP 405 is aimed towards a rather different goal from yours - to be able to share an installed Python's copy of the standard library among a number of virtual environments. This is the purpose of the "home" key in pyvenv.cfg and the added sys.base_prefix (which point to the Python installation whose standard library will be used). I think this serves a valuable and common use case, but I wonder if your use case couldn't also be served with a minor tweak to PEP 405. Currently it ignores a pyvenv.cfg file with no "home" key; instead, it could set sys.prefix and sys.base_prefix both to the location of that pyvenv.cfg. For most purposes, this would result in a broken Python (no standard library), but it might help you?
Instead of no home key, how about an empty home key? Explicit being better, and all that. ~Ethan~
-----Original Message----- From: Carl Meyer [mailto:carl@oddbird.net] Sent: 19. mars 2012 19:19 To: Kristján Valur Jónsson Cc: Python-Dev (python-dev@python.org) Subject: Re: [Python-Dev] PEP 405 (built-in virtualenv) status
Hello Kristján, I think there's one important (albeit odd and magical) bit of Python's current behavior that you are missing in your blog post. All of the initial sys.path directories are constructed relative to sys.prefix and sys.exec_prefix, and those values in turn are determined (if PYTHONHOME is not set), by walking up the filesystem tree from the location of the Python binary, looking for the existence of a file at the relative path "lib/pythonX.X/os.py" (or "Lib/os.py" on Windows). Python takes the existence of this file to mean that it's found the standard library, and sets sys.prefix accordingly. Thus, you can achieve reliable full isolation from any installed Python, with no need for environment variables, simply by placing a file (it can even be empty) at that relative location from the location of your Python binary. You will still get some default paths added on sys.path, but they will all be relative to your Python binary and thus presumably under your control; nothing from any other location will be on sys.path. I doubt you will find this solution satisfyingly elegant, but you might nonetheless find it practically useful.
Right. Thanks for explaining this. Although, it would appear that Python also has a mechanism for detecting that it is being run from a build environment and ignore PYTHONHOME in that case too.
Beyond that possible tweak, while I certainly wouldn't oppose any effort to clean up / document / make-optional Python's startup sys.path-setting behavior, I think it's mostly orthogonal to PEP 405, and I don't think it would be helpful to expand the scope of PEP 405 to include that effort.
Well, it sounds as this pep can definitely be used as the basis for work to completely customize the startup behaviour. In my case, it would be desirable to be able to completely ignore any PYTHONHOME environment variable (and any others). I'd also like to be able to manually set up the sys.path. Perhaps if we can set things up that one key (ignore_env) will cause the environment variables to be ignored, and then, an empty home key will set the sys.path to point to the directory of the .cfg file. Presumably, this would then cause a site.py found at that place to be executed and one could code whatever extra logic one wants into that file. Possibly a "site" key in the .cfg file would achieve the same goal, allowing the user to call this setup file whatever he wants. With something like this in place, the built in behaviour of python.exe to realize that it is running from a "build" environment and in that case ignore PYTHONPATH and set a special sys.path, could all be removed from being hardcoded into being coded into some buildsite.py in the cpython root folder. Kristján
On 21.03.12 14:35, Kristján Valur Jónsson wrote:
-----Original Message----- From: Carl Meyer [mailto:carl@oddbird.net] Sent: 19. mars 2012 19:19 To: Kristján Valur Jónsson Cc: Python-Dev (python-dev@python.org) Subject: Re: [Python-Dev] PEP 405 (built-in virtualenv) status
Hello Kristján, I think there's one important (albeit odd and magical) bit of Python's current behavior that you are missing in your blog post. All of the initial sys.path directories are constructed relative to sys.prefix and sys.exec_prefix, and those values in turn are determined (if PYTHONHOME is not set), by walking up the filesystem tree from the location of the Python binary, looking for the existence of a file at the relative path "lib/pythonX.X/os.py" (or "Lib/os.py" on Windows). Python takes the existence of this file to mean that it's found the standard library, and sets sys.prefix accordingly. Thus, you can achieve reliable full isolation from any installed Python, with no need for environment variables, simply by placing a file (it can even be empty) at that relative location from the location of your Python binary. You will still get some default paths added on sys.path, but they will all be relative to your Python binary and thus presumably under your control; nothing from any other location will be on sys.path. I doubt you will find this solution satisfyingly elegant, but you might nonetheless find it practically useful.
Right. Thanks for explaining this. Although, it would appear that Python also has a mechanism for detecting that it is being run from a build environment and ignore PYTHONHOME in that case too.
Beyond that possible tweak, while I certainly wouldn't oppose any effort to clean up / document / make-optional Python's startup sys.path-setting behavior, I think it's mostly orthogonal to PEP 405, and I don't think it would be helpful to expand the scope of PEP 405 to include that effort. Well, it sounds as this pep can definitely be used as the basis for work to completely customize the startup behaviour. In my case, it would be desirable to be able to completely ignore any PYTHONHOME environment variable (and any others). I'd also like to be able to manually set up the sys.path.
Perhaps if we can set things up that one key (ignore_env) will cause the environment variables to be ignored, and then, an empty home key will set the sys.path to point to the directory of the .cfg file. Presumably, this would then cause a site.py found at that place to be executed and one could code whatever extra logic one wants into that file. Possibly a "site" key in the .cfg file would achieve the same goal, allowing the user to call this setup file whatever he wants.
With something like this in place, the built in behaviour of python.exe to realize that it is running from a "build" environment and in that case ignore PYTHONPATH and set a special sys.path, could all be removed from being hardcoded into being coded into some buildsite.py in the cpython root folder.
As an old windows guy, I very much agree with Kristjan. The venv approach is great. Windows is just a quite weird situation to handle in some cases, and a super-simple way to get rid of *any* built-in behavior concerning setup would be great. The idea of moving path setup stuff into the python.exe stub makes very much sense to me. This would make pythonxx.dll a really useful library to be shared. Kristjan can then provide his own custom python.exe and be assured the python dll will not try to lurk into something unforeseen. I think this would also be a security aspect: The dll can be considered really safe for sandboxing if it does not even have the ability to change the python behavior by built-in magic. Besides that, I agree with Ethan that explicit is better than implicit, again. I am missing even more explicitness: Python has IMHO too much behavior like this: 'by default, look into xxx, but if a yyy exists, behave differently'. I don't like this, because the absense of a simple file changes the whole system behavior. I would do it the other way round: As soon as you introduce the venv.cfg file, enforce its existence completely! If that file is not there, then python exits with an error message. This way you can safely ensure its existence, and the file can be made read-only and so on. A non-existent file is just a bad thing and is hard to make read-only ;-) So please let's abandon the old 'if exists ...' pattern, at least this one time. By the explicit cfg file, the file can clearly say if there is a virtual env or not. Together with removing magic from the .dll, the situation at least for windows would greatly improve. ciao - chris -- Christian Tismer :^)<mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a. PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Sun, Jun 3, 2012 at 3:33 AM, Christian Tismer <tismer@stackless.com> wrote:
As an old windows guy, I very much agree with Kristjan. The venv approach is great. Windows is just a quite weird situation to handle in some cases, and a super-simple way to get rid of *any* built-in behavior concerning setup would be great.
The idea of moving path setup stuff into the python.exe stub makes very much sense to me. This would make pythonxx.dll a really useful library to be shared.
It's mainly Py_Initialize() that triggers the magic. What may be worth exploring is a variant on that which allows embedding applications to explicitly pass in *everything* that would otherwise be guessed by inspecting the environment. (Some things can be forced to particular values already, but certainly not everything).
Python has IMHO too much behavior like this: 'by default, look into xxx, but if a yyy exists, behave differently'. I don't like this, because the absense of a simple file changes the whole system behavior. I would do it the other way round: As soon as you introduce the venv.cfg file, enforce its existence completely! If that file is not there, then python exits with an error message. This way you can safely ensure its existence, and the file can be made read-only and so on. A non-existent file is just a bad thing and is hard to make read-only ;-) So please let's abandon the old 'if exists ...' pattern, at least this one time. By the explicit cfg file, the file can clearly say if there is a virtual env or not.
Backwards compatibility constraints mean we simply can't do that. However, as noted above, it may make sense to provide more ways for embedding applications to selectively access the behaviour through the C API. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 04.06.12 00:28, Nick Coghlan wrote:
... Backwards compatibility constraints mean we simply can't do that. However, as noted above, it may make sense to provide more ways for embedding applications to selectively access the behaviour through the C API.
Why that??? I don't see this. If you have a new python version with a new file that has-to-be-there, what is then the problem? The new version carries the new file, so I don't see a compatibility issue, because this version does not want to be backward-compatible. It just introduces the new file constraint, and it produces what it needs. Am I somehow blinded, maybe? (yes, you all know that I am, so please be patient with me) -- Chris -- Christian Tismer :^)<mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a. PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
participants (6)
-
Carl Meyer -
Christian Tismer -
Ethan Furman -
Glenn Linderman -
Kristján Valur Jónsson -
Nick Coghlan