Use ptyhon -s as default shbang for system python executables/daemons

We're starting a discussion in Fedora about setting the default shbang for system python executables and/or daemons to python -s or python -Es (or ?). See also: https://bugzilla.redhat.com/show_bug.cgi?id=1202151 https://fedorahosted.org/fpc/ticket/513 Basically we're wanting to avoid locally installed items causing security issues or other bad behavior, without too adversely affecting users' abilities to work around issues or intentionally alter behavior. It would be good to get some feedback from the broader python community before implementing anything, so I'm asking for feedback here. Thank you, Orion

On Mar 18, 2015, at 03:46 PM, Orion Poplawski wrote:
We're starting a discussion in Fedora about setting the default shbang for system python executables and/or daemons to python -s or python -Es (or ?).
We've talked about this in Debian/Ubuntu land and the general consensus is that for Python 2, use -Es and for Python 3 use -I (which implies -Es). I'm not sure we're consistent yet in making sure our build tools install these switches in our shebangs, but I'm hoping after Jessie we can make some progress on that. Cheers, -Barry

On Wed, Mar 18, 2015 at 12:22:03PM -0400, Barry Warsaw wrote:
On Mar 18, 2015, at 03:46 PM, Orion Poplawski wrote:
We're starting a discussion in Fedora about setting the default shbang for system python executables and/or daemons to python -s or python -Es (or ?).
We've talked about this in Debian/Ubuntu land and the general consensus is that for Python 2, use -Es and for Python 3 use -I (which implies -Es). I'm not sure we're consistent yet in making sure our build tools install these switches in our shebangs, but I'm hoping after Jessie we can make some progress on that.
Interesting, I've cautiously in favor of -s in Fedora but the more I've thought about it the less I've liked -E. It just seems like PYTHONPATH is analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting. We leave both of those for local admins and users to affect the behaviour of programs if they needed to. Was there some discussion of -E specifically in Debian where it was consciously decided that PYTHONPATH was not analagous or that the benefit risk was different than for those other env vars? -Toshio

On Mar 18, 2015, at 02:44 PM, Toshio Kuratomi wrote:
Interesting, I've cautiously in favor of -s in Fedora but the more I've thought about it the less I've liked -E. It just seems like PYTHONPATH is analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting. We leave both of those for local admins and users to affect the behaviour of programs if they needed to.
It is, and it isn't. It's different because you can always explicitly override the shebang line if needed. So if a local admin really needed to override $PYTHONPATH (though I can't come up with a use case right now), they could just: $ python3 -s /usr/bin/foo
Was there some discussion of -E specifically in Debian where it was consciously decided that PYTHONPATH was not analagous or that the benefit risk was different than for those other env vars?
I'd have to go digging around the archives. It wasn't a recent discussion IIRC. Cheers, -Barry

On Wed, Mar 18, 2015 at 2:56 PM, Barry Warsaw <barry@python.org> wrote:
On Mar 18, 2015, at 02:44 PM, Toshio Kuratomi wrote:
Interesting, I've cautiously in favor of -s in Fedora but the more I've thought about it the less I've liked -E. It just seems like PYTHONPATH is analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting. We leave both of those for local admins and users to affect the behaviour of programs if they needed to.
It is, and it isn't. It's different because you can always explicitly override the shebang line if needed. So if a local admin really needed to override $PYTHONPATH (though I can't come up with a use case right now), they could just:
$ python3 -s /usr/bin/foo
I could see that as a difference. However, the environment variables give users the ability to change things globally whereas overriding the shebang line is case-by-case so it's not a complete replacement of the functionality. LD_LIBRARY_PATH can be used for things like logging all calls to a specific function, applying a bugfix to a library when you don't have root on the box, evaluating how a potential replacement for a system library will affect the whole system, and other things that are supposed to affect a large number of the files on the OS. PYTHONPATH can be used for the same purposes as long as -E is not embedded into the shebang lines. (I suppose you could write a "python" wrapper script that discards -E... but you'd need root on the box to install your wrapper [since system packages are encouraged to use the full path to python rather than env python] and the change would affect everyone on the box rather than just the person setting the env var). Using -E by default for all system applications would prevent that. The benefit of -E is that it isolates the effects of PYTHONPATH to only specific programs. However, that benefit doesn't seem as striking as it first appears (or at least, as it first appeared to me :-). Unix env vars have their own method of isolation: if the env var is marked for export then it is sent to child processes. If it is not marked for export then it only affects the current process. So it seems like -E isn't adding something new; it's just protecting users from themselves. That seems contrary to "consenting adults" (although distributions are separate entities from python-dev ;-). What makes -s different from -E? If you think about it in the context of users being able to set PYTHONPATH to add libraries that can override system packages then I think specifying -s for system packages establishes a default behaviour: The user can override the defaults but only if they change the environment. Without -s, this expectation is violated for libraries in the user site directory. With -s, the user would have to add the user site directory to PYTHONPATH if they want the libraries there to override system packages. So I guess I'm still leaning towards -E being the wrong choice for Fedora but Fedora lives within a broader ecosystem of python-providing distributions. So I'm interested in seeing whether Debian thought about these aspects when they decided on using -E or if that would change anyone's minds and also what other distributions think about adding -s and/or -E to their packaged applications' shebang lines. -Toshio

On Mar 19, 2015, at 11:15 AM, Toshio Kuratomi wrote:
I could see that as a difference. However, the environment variables give users the ability to change things globally whereas overriding the shebang line is case-by-case so it's not a complete replacement of the functionality.
You make some good points. I guess it's a trade-off between flexibility and a known secure execution environment. I'm not sure there's a right answer; different admins might have valid different opinions. Cheers, -Barry

I think I've found the Debian discussion (October 2012): http://comments.gmane.org/gmane.linux.debian.devel.python/8188 Lack of PYTHONWARNINGS was brought up late in the discussion thread but I think the understanding that when a particular user sets an environment variable they want it to apply to all scripts they run was kind of lost in the followups (It wasn't directly addressed or mentioned again.) -Toshio On Thu, Mar 19, 2015 at 12:27 PM, Barry Warsaw <barry@python.org> wrote:
On Mar 19, 2015, at 11:15 AM, Toshio Kuratomi wrote:
I could see that as a difference. However, the environment variables give users the ability to change things globally whereas overriding the shebang line is case-by-case so it's not a complete replacement of the functionality.
You make some good points. I guess it's a trade-off between flexibility and a known secure execution environment. I'm not sure there's a right answer; different admins might have valid different opinions.
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/a.badger%40gmail.com

2015-03-19 21:47 GMT+01:00 Toshio Kuratomi <a.badger@gmail.com>:
I think I've found the Debian discussion (October 2012):
http://comments.gmane.org/gmane.linux.debian.devel.python/8188
Lack of PYTHONWARNINGS was brought up late in the discussion thread
Maybe we need to modify -E or add a new option to only ignore PYTHONPATH. Victor

-Toshio On Mar 19, 2015 3:27 PM, "Victor Stinner" <victor.stinner@gmail.com> wrote:
2015-03-19 21:47 GMT+01:00 Toshio Kuratomi <a.badger@gmail.com>:
I think I've found the Debian discussion (October 2012):
http://comments.gmane.org/gmane.linux.debian.devel.python/8188
Lack of PYTHONWARNINGS was brought up late in the discussion thread
Maybe we need to modify -E or add a new option to only ignore PYTHONPATH.
I think pythonpath is still useful on its own. Building off Nick's idea of a system python vs a python for users to use, I would see a more useful modification to be able to specify SPYTHONPATH (and other env vars) to go along with /usr/bin/spython . That way the user maintains the capability to override specific libraries globally just like with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally configure your own python to use one set of paths for your five python apps and then have that leak over and affect system tools. -Toshio

On Mon, 23 Mar 2015 07:22:56 -0700 Toshio Kuratomi <a.badger@gmail.com> wrote:
Building off Nick's idea of a system python vs a python for users to use, I would see a more useful modification to be able to specify SPYTHONPATH (and other env vars) to go along with /usr/bin/spython . That way the user maintains the capability to override specific libraries globally just like with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally configure your own python to use one set of paths for your five python apps and then have that leak over and affect system tools.
I really think Donald has a good point when he suggests a specific virtualenv for system programs using Python. Regards Antoine.

On Mon, Mar 23, 2015 at 03:30:23PM +0100, Antoine Pitrou wrote:
On Mon, 23 Mar 2015 07:22:56 -0700 Toshio Kuratomi <a.badger@gmail.com> wrote:
Building off Nick's idea of a system python vs a python for users to use, I would see a more useful modification to be able to specify SPYTHONPATH (and other env vars) to go along with /usr/bin/spython . That way the user maintains the capability to override specific libraries globally just like with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally configure your own python to use one set of paths for your five python apps and then have that leak over and affect system tools.
I really think Donald has a good point when he suggests a specific virtualenv for system programs using Python.
The isolation is what we're seeking but I think the amount of work required and the added complexity for the distributions will make that hard to get distributions to sign up for. If someone had the time to write a front end to install packages into a single "system-wide isolation unit" whose backend was a virtualenv we might be able to get distributions on-board with using that. The front end would need to install software so that you can still invoke /usr/bin/system-application and "system-application" would take care of activating the virtualenv. It would need to be about as simple to build as the present python2 setup.py build/install with the flexibility in options that the distros need to install into FHS approved paths. Some things like man pages, locale files, config files, and possibly other data files might need to be installed outside of the virtualenv directory. Many setup.py's already punt on some of those, though, letting the user choose to install them manually. So this might be similar. It would need to be able to handle 32bit and 64bit versions of the same library installed on the same system. It would need to be able to handle different versions of the same library installed on the same system (as few of those as possible but it's an unfortunate cornercase that can't be entirely ignored even for just system packages). It would need a mode where it doesn't use the network at all; only operates with the packages and sources that are present on the box. And remember these two things: (1) we'd be asking the distros to do a tremendous amount of work changing their packages to install into a virtualenv instead of the python setup.py way that is well documented and everyone's been using for ages. it'll be a tough sell even with good tooling. (2) this theoretical front-end would have to appeal to the distro maintainers so there would have to be a lot of talk to understand what capabilities the distro maintainers would need from it. -Toshio

On Mon, 23 Mar 2015 08:06:13 -0700 Toshio Kuratomi <a.badger@gmail.com> wrote:
I really think Donald has a good point when he suggests a specific virtualenv for system programs using Python.
The isolation is what we're seeking but I think the amount of work required and the added complexity for the distributions will make that hard to get distributions to sign up for.
If someone had the time to write a front end to install packages into a single "system-wide isolation unit" whose backend was a virtualenv we might be able to get distributions on-board with using that.
I don't think we're asking distributions anything. We're suggesting a possible path, but it's not python-dev's job to dictate distributions how they should package Python. The virtualenv solution has the virtue that any improvement we might put in it to help system packagers would automatically benefit everyone. A specific "system Python" would not.
The front end would need to install software so that you can still invoke /usr/bin/system-application and "system-application" would take care of activating the virtualenv. It would need to be about as simple to build as the present python2 setup.py build/install with the flexibility in options that the distros need to install into FHS approved paths. Some things like man pages, locale files, config files, and possibly other data files might need to be installed outside of the virtualenv directory.
Well, I don't understand what difference a virtualenv would make. Using a virtualenv amounts to invoking a different interpreter path. The rest of the filesystem (man pages locations, etc.) is still accessible in the same way. But I may miss something :-) Regards Antoine.

On Mar 23, 2015 8:15 AM, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
On Mon, 23 Mar 2015 08:06:13 -0700 Toshio Kuratomi <a.badger@gmail.com> wrote:
I really think Donald has a good point when he suggests a specific virtualenv for system programs using Python.
The isolation is what we're seeking but I think the amount of work
required
and the added complexity for the distributions will make that hard to get distributions to sign up for.
If someone had the time to write a front end to install packages into a single "system-wide isolation unit" whose backend was a virtualenv we might be able to get distributions on-board with using that.
I don't think we're asking distributions anything. We're suggesting a possible path, but it's not python-dev's job to dictate distributions how they should package Python.
The virtualenv solution has the virtue that any improvement we might put in it to help system packagers would automatically benefit everyone. A specific "system Python" would not.
The front end would need to install software so that you can still invoke /usr/bin/system-application and "system-application" would take care of activating the virtualenv. It would need to be about as simple to build as the present python2 setup.py build/install with the flexibility in options that the distros need to install into FHS approved paths. Some things like man pages, locale files, config files, and possibly other data files might need to be installed outside of the virtualenv directory.
Well, I don't understand what difference a virtualenv would make. Using a virtualenv amounts to invoking a different interpreter path. The rest of the filesystem (man pages locations, etc.) is still accessible in the same way. But I may miss something :-)
The main issue that jumps to my mind is that 'yum/apt-get install some-python-package' should install it into both the base python interpreter and the system virtualenv, but that 'sudo pip install some-python-package' should install into only the base interpreter but not the system virtualenv. (Even if those two commands are run in sequence with different versions of some-python-package.) This seems potentially complex. -n

On Mon, Mar 23, 2015 at 04:14:52PM +0100, Antoine Pitrou wrote:
On Mon, 23 Mar 2015 08:06:13 -0700 Toshio Kuratomi <a.badger@gmail.com> wrote:
I really think Donald has a good point when he suggests a specific virtualenv for system programs using Python.
The isolation is what we're seeking but I think the amount of work required and the added complexity for the distributions will make that hard to get distributions to sign up for.
If someone had the time to write a front end to install packages into a single "system-wide isolation unit" whose backend was a virtualenv we might be able to get distributions on-board with using that.
I don't think we're asking distributions anything. We're suggesting a possible path, but it's not python-dev's job to dictate distributions how they should package Python.
The virtualenv solution has the virtue that any improvement we might put in it to help system packagers would automatically benefit everyone. A specific "system Python" would not.
The front end would need to install software so that you can still invoke /usr/bin/system-application and "system-application" would take care of activating the virtualenv. It would need to be about as simple to build as the present python2 setup.py build/install with the flexibility in options that the distros need to install into FHS approved paths. Some things like man pages, locale files, config files, and possibly other data files might need to be installed outside of the virtualenv directory.
Well, I don't understand what difference a virtualenv would make. Using a virtualenv amounts to invoking a different interpreter path. The rest of the filesystem (man pages locations, etc.) is still accessible in the same way. But I may miss something :-)
<nod> I think people who are saying "The system should just use virtualenv" aren't realizing all of the reasons that's not as simple as it sounds for distributions to implement. thus the work required to implement alternate solutions like a system python may seem less to the distros unless those issues are partially addressed at the virtualenv and python-packaging level. -Toshio

2015-03-18 16:46 GMT+01:00 Orion Poplawski <orion@cora.nwra.com>:
We're starting a discussion in Fedora about setting the default shbang for system python executables and/or daemons to python -s or python -Es (or ?).
Python 3.4 has -I which is more strict than -Es. It remembers me "Perl suid", /usr/bin/sperl. Maybe we should have a "spython" program which is like "python -I" (so it means adding spython, spython3, spython3.5). Does it work to pass command line options to Python in the shebang?
Basically we're wanting to avoid locally installed items causing security issues or other bad behavior, without too adversely affecting users' abilities to work around issues or intentionally alter behavior.
It would be good to get some feedback from the broader python community before implementing anything, so I'm asking for feedback here.
Personally, instead of having to enable a switch to have a safe Python, I would prefer to have a safe Python by default and enable an option to enter the unsafe mode. But it may break backward compatibility depending on changes made in Python if we take this way. Victor

On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
Does it work to pass command line options to Python in the shebang?
Yes, but only one "word", thus -Es or -I. We've often mused about whether it makes sense to have two Pythons on the system. One for system scripts and another for users. System Python ('/usr/bin/spython' perhaps) would be locked down and only extensible by system packages. On Debuntu that might mean no /usr/local on sys.path. It would also have a much more limited set of installed packages, i.e. only those needed to support system functionality. /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the goodness they currently have. It's never gotten much farther than musings, but protecting the system against the weird things people install would be a good thing. OTOH, this feels a lot like virtual environments so maybe there's something useful to be mined there. Cheers, -Barry

On Mar 18, 2015, at 12:48 PM, Barry Warsaw <barry@python.org> wrote:
On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
Does it work to pass command line options to Python in the shebang?
Yes, but only one "word", thus -Es or -I.
We've often mused about whether it makes sense to have two Pythons on the system. One for system scripts and another for users. System Python ('/usr/bin/spython' perhaps) would be locked down and only extensible by system packages. On Debuntu that might mean no /usr/local on sys.path. It would also have a much more limited set of installed packages, i.e. only those needed to support system functionality.
/usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the goodness they currently have.
It's never gotten much farther than musings, but protecting the system against the weird things people install would be a good thing. OTOH, this feels a lot like virtual environments so maybe there's something useful to be mined there.
Cheers, -Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
I’ve long wished that the OS had it’s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mar 18, 2015, at 05:51 PM, Donald Stufft wrote:
I’ve long wished that the OS had it’s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace.
Yep, and those breakages can be difficult to debug. Comment #47 for Bug 123456: "Oh wait, you installed *what?!*" Cheers, -Barry

On 19 March 2015 at 07:51, Donald Stufft <donald@stufft.io> wrote:
I’ve long wished that the OS had it’s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace.
I'm more wanting to go in the other direction and suggest to folks that if they're not *scripting the OS*, then the system Python isn't what they want, and they should be using at least a virtual environment, preferably an entirely separate software collection that they can upgrade on their own timeline, independently of what they system does. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, 21 Mar 2015 21:52:34 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 19 March 2015 at 07:51, Donald Stufft <donald@stufft.io> wrote:
I’ve long wished that the OS had it’s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace.
I'm more wanting to go in the other direction and suggest to folks that if they're not *scripting the OS*, then the system Python isn't what they want, and they should be using at least a virtual environment,
So that defeats the whole point of having a system distribution. I don't have any stakes in Debian / RedHat etc., but that sounds a little bit ridiculous to me ;-) On the other hand, perhaps it would allow ditching any requests to keep code compatible with old versions such as 2.6. Regards Antoine.

On 21 March 2015 at 22:19, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 21 Mar 2015 21:52:34 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 19 March 2015 at 07:51, Donald Stufft <donald@stufft.io> wrote:
I’ve long wished that the OS had it’s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace.
I'm more wanting to go in the other direction and suggest to folks that if they're not *scripting the OS*, then the system Python isn't what they want, and they should be using at least a virtual environment,
So that defeats the whole point of having a system distribution.
Given how many of the operating system packages are written in Python, it really doesn't - system administrators getting to use Python to script the OS is a secondary purpose, the system Python installation is mostly there to make the OS itself work (if it wasn't for that, migrating the system Python to Python 3 would have been a lot simpler!). In my view, running system independent applications in the system Python is at best a tertiary purpose - many of those apps (especially web services) *don't want* to be using system modules at all, which is why virtualenv switched to disabling the global site-packages by default.
I don't have any stakes in Debian / RedHat etc., but that sounds a little bit ridiculous to me ;-) On the other hand, perhaps it would allow ditching any requests to keep code compatible with old versions such as 2.6.
That's exactly the reason for the preference - unless they're specifically wanting to script RHEL/CentOS 6 or a similar long term support version at this point, ideally folks should be able to use Python 2.7 as their baseline version by now. RHEL/CentOS 7 are both on 2.7, while Python 2.7 has been available for RHEL/CentOS 6 by way of software collections for a couple of years now. When Python 2 goes EOL upstream in 2020, the standard 10 years of support on RHEL 7 will still have 4 years left to run - I'd prefer to see most folks migrated to a Python 3 software collection by then (or failing that, at least to the Python 3 stack Slavek is aiming to get into EPEL 7). Unfortunately, as far as I am aware, Debian Stable and Ubuntu LTS don't yet have a distro-endorsed equivalent of softwarecollections.org, and it's not yet clear if running on a faster moving base distro in a Docker or Rocket container will be an adequate alternative. I believe SCLs moved over to using Linux environment modules as the backend tech in 2.0 (for compatibility with existing HPC environments), so it should likely be possible to adapt them to work with APT in addition to RPM, but that requires finding folks both interested in doing the technical work, as well as in making the case for SCLs as a feature in the relevant distros (they're a large enough departure from the traditional mode of operation of a Linux distro that it took a long time to find a stable upstream home for the ongoing maintenance of SCLs for the Fedora/RHEL/CentOS ecosystem - primary maintenance responsibility finally landed earlier this year as a CentOS SIG) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mar 21, 2015, at 7:52 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 19 March 2015 at 07:51, Donald Stufft <donald@stufft.io> wrote:
I’ve long wished that the OS had it’s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace.
I'm more wanting to go in the other direction and suggest to folks that if they're not *scripting the OS*, then the system Python isn't what they want, and they should be using at least a virtual environment, preferably an entirely separate software collection that they can upgrade on their own timeline, independently of what they system does.
It’s likely easier to get the OS to move it’s own things to a virtual environment than it is to convince every single person who uses an OS to never install globally. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 21Mar2015 14:29, Donald Stufft <donald@stufft.io> wrote:
On Mar 21, 2015, at 7:52 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: On 19 March 2015 at 07:51, Donald Stufft <donald@stufft.io> wrote:
I’ve long wished that the OS had it’s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace.
I'm more wanting to go in the other direction and suggest to folks that if they're not *scripting the OS*, then the system Python isn't what they want, and they should be using at least a virtual environment, preferably an entirely separate software collection that they can upgrade on their own timeline, independently of what they system does.
It’s likely easier to get the OS to move it’s own things to a virtual environment than it is to convince every single person who uses an OS to never install globally.
I agree. And just as a data point, this cropped up on the Fedora list yesterday: I broke Yum (by messing with Python libs) http://www.spinics.net/linux/fedora/fedora-users/msg458069.html TL;DR: OP used pip on his system python. Yum broke. Probably hampered his attempts to repair, too. Cheers, Cameron Simpson <cs@zip.com.au> The mind reigns, but does not govern. - Paul Valery

On 19 March 2015 at 02:48, Barry Warsaw <barry@python.org> wrote:
It's never gotten much farther than musings, but protecting the system against the weird things people install would be a good thing. OTOH, this feels a lot like virtual environments so maybe there's something useful to be mined there.
I took it a fair bit further than musings: https://www.python.org/dev/peps/pep-0432/#a-system-python-executable Unfortunately PEP 432's a big project with very high barriers to entry (you need to come to grips with the current arcane startup sequence first), so I've never managed to hand it over after running out of time to pursue it myself. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw <barry@python.org> wrote:
On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
Does it work to pass command line options to Python in the shebang?
Yes, but only one "word", thus -Es or -I.
We've often mused about whether it makes sense to have two Pythons on the system. One for system scripts and another for users. System Python ('/usr/bin/spython' perhaps) would be locked down and only extensible by system packages. On Debuntu that might mean no /usr/local on sys.path. It would also have a much more limited set of installed packages, i.e. only those needed to support system functionality.
I recommend this. It'd be nice to see a common OS distro actually do it. For a system-only Python lockdown you should remove distutils and pip to prevent anyone from easily installing anything that isn't a distro package into it. Possibly even removing its site-packages directory and sys.path entry all together (things your distro needs to install could mix directly with the stdlib packages)
/usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the goodness they currently have.
It's never gotten much farther than musings, but protecting the system against the weird things people install would be a good thing. OTOH, this feels a lot like virtual environments so maybe there's something useful to be mined there.
While people sometimes suggest virtualenv as a solution for this. It isn't really the same thing. It isn't a hermetic clone of the original interpreter. It copies the main binary but symlinks back to much of the stdlib. So any existing virtualenv can be an out of date unsupported mix of both after the original interpreter is updated. This has caused users problems in the past with some minor version updates where an internal non-public API used between some binary and a stdlib module was updated as part of a bugfix. Suddenly they didn't match up for existing virtualenvs. virtualenv is an amazing hack that I promote to most anyone for their own applications use with the occasional known caveats (solvable by regurly rebuilding your virtualenvs)... But I wouldn't want to see it used for the core OS itself even though it sounds better at first glance. -gps

On Mon, 23 Mar 2015 21:58:06 +0000 "Gregory P. Smith" <greg@krypto.org> wrote:
virtualenv is an amazing hack that I promote to most anyone for their own applications use with the occasional known caveats (solvable by regurly rebuilding your virtualenvs)... But I wouldn't want to see it used for the core OS itself even though it sounds better at first glance.
pyvenv is not a hack, it mostly creates a couple symlinks and adds a pip install into the environment. Regards Antoine.

On 23Mar2015 21:58, Gregory P. Smith <greg@krypto.org> wrote:
While people sometimes suggest virtualenv as a solution for this. It isn't really the same thing. It isn't a hermetic clone of the original interpreter. It copies the main binary but symlinks back to much of the stdlib.
Oh. I had thought a non-standalone venv arranged sys.path to fall back to the source interpreter. Clearly I have not paid attention. Cheers, Cameron Simpson <cs@zip.com.au> Yes, sometimes Perl looks like line-noise to the uninitiated, but to the seasoned Perl programmer, it looks like checksummed line-noise with a mission in life. - The Llama Book

On 24 Mar 2015 07:59, "Gregory P. Smith" <greg@krypto.org> wrote:
On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw <barry@python.org> wrote:
On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
Does it work to pass command line options to Python in the shebang?
Yes, but only one "word", thus -Es or -I.
We've often mused about whether it makes sense to have two Pythons on the system. One for system scripts and another for users. System Python ('/usr/bin/spython' perhaps) would be locked down and only extensible by system packages. On Debuntu that might mean no /usr/local on sys.path.
would also have a much more limited set of installed packages, i.e. only
needed to support system functionality.
I recommend this. It'd be nice to see a common OS distro actually do it.
For a system-only Python lockdown you should remove distutils and pip to
/usr/bin/python2 and /usr/bin/python3 then would be user tools, with all
goodness they currently have.
It's never gotten much farther than musings, but protecting the system against the weird things people install would be a good thing. OTOH, this feels a lot like virtual environments so maybe there's something useful to be mined
It those prevent anyone from easily installing anything that isn't a distro package into it. Possibly even removing its site-packages directory and sys.path entry all together (things your distro needs to install could mix directly with the stdlib packages) The hard part of this is the complexity in getpath.c and the current difficulty of testing the interaction of all the different configuration settings properly. Completely overriding the standard path setup isn't too bad, it's *partially* overriding it that's hard. That's really one of the main goals of the proposed startup refactoring in PEP 432 - separating things into distinct phases so the path configuration is easier to customise and so the current behaviour becomes easier to test. That's a fairly big project with high barriers to entry though, as even just wrapping your head around the current accumulated complexity in the way the default sys.path gets constructed is a time consuming task. That approach has the virtue of not being a "system Python" specific solution, it's a "make embedding the CPython runtime much easier" solution, with "system Python with different default settings" as the reference use case. the there. For distro level work it would be closer to the environment modules used in high performance computing work (which are also the basis of softwarecollections.org) Where things get really messy is on the package management side. Either we end up with yet another copy of modules that are often already duplicated between the Python 2 & 3 stacks, or else the relevant user Python needs to be customised to also be able to see the system level libraries for things like Kerberos, interfacing with the package manager, dbus, etc. The former approach = "ugh, no, just no" as far as I'm concerned, while the latter would be a messy patch to try to carry given the current difficulty of customising and testing the startup sequence.
While people sometimes suggest virtualenv as a solution for this. It
isn't really the same thing. It isn't a hermetic clone of the original interpreter. It copies the main binary but symlinks back to much of the stdlib. So any existing virtualenv can be an out of date unsupported mix of both after the original interpreter is updated. This has caused users problems in the past with some minor version updates where an internal non-public API used between some binary and a stdlib module was updated as part of a bugfix. Suddenly they didn't match up for existing virtualenvs.
virtualenv is an amazing hack that I promote to most anyone for their own
applications use with the occasional known caveats (solvable by regurly rebuilding your virtualenvs)... But I wouldn't want to see it used for the core OS itself even though it sounds better at first glance. Right, this is why Fedora/RHEL/CentOS have software collections instead, and those have their own "gotchas" that currently make them unsuitable for use as the main system Python (for example: the Fedora installer & package manager are both written in Python, so having the main system Python be an SCL would bring SCL support into the minimal package set, which we wouldn't want to do. The typical aim is to find ways to get things *out* of that set, not add new ones) Cheers, Nick.

Orion Poplawski <orion@cora.nwra.com> wrote:
It would be good to get some feedback from the broader python community before implementing anything, so I'm asking for feedback here.
On my systems I have /use/bin/python for the system and ~/anaconda/python for me. Apple and Ubuntu can do whatever they want with their Python's, I am not touching them. The main thing is to keep the system Python and the user Python separate. That is, both the executable and the folder where packages are installed. Whatever I install for myself can fuck up ~/anaconda, but is not allowed to mess with the system files or system folders. Sturla

On 19Mar2015 19:57, Sturla Molden <sturla.molden@gmail.com> wrote:
Orion Poplawski <orion@cora.nwra.com> wrote:
It would be good to get some feedback from the broader python community before implementing anything, so I'm asking for feedback here.
On my systems I have /use/bin/python for the system and ~/anaconda/python for me. Apple and Ubuntu can do whatever they want with their Python's, I am not touching them. The main thing is to keep the system Python and the user Python separate. That is, both the executable and the folder where packages are installed. Whatever I install for myself can fuck up ~/anaconda, but is not allowed to mess with the system files or system folders.
Me too. I keep my default virtualenvs in ~/var/venv, and have personal "py2" and "py3" scripts that invoke via the venvs. Having a similar separation in the system would be a good thing, for the same reasons. Also, it would let the OS supplier keep a _much_ smaller package/addon list for the "core" admin python instance, making QA etc easier and more reliable. Popular OSes let the local sysadmin (== "the user") pull in all sorts of stuff to the "system" Python, even from the supplier's repositories. Having a walled off "core" admin python as well seems very prudent. Cheers, Cameron Simpson <cs@zip.com.au> conclude that this language as a tool is an open invitation for clever tricks; and while exactly this may be the explanation for some of its appeal, /viz./ to those who like to show how clever they are. I am sorry, but I must regard this as one of the most damning things that can be said about a programming language. - Edsger Dijkstra, _The Humble Programmer_ (CACM, Ocotber, 1972)
participants (11)
-
Antoine Pitrou
-
Barry Warsaw
-
Cameron Simpson
-
Donald Stufft
-
Gregory P. Smith
-
Nathaniel Smith
-
Nick Coghlan
-
Orion Poplawski
-
Sturla Molden
-
Toshio Kuratomi
-
Victor Stinner