Have virtual environments led to neglect of the actual environment?
![](https://secure.gravatar.com/avatar/3dd475b8aaa5292d74cb0c3f76c3f196.jpg?s=120&d=mm&r=g)
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking: Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly? I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason. How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?
![](https://secure.gravatar.com/avatar/833c1ef94aebf738851577a5cb0b71e7.jpg?s=120&d=mm&r=g)
I think it's a classic case of dependency hell. OS packagers are rebundling Python packages as OS packages and expressing their own OS-package dependency graphs. Then, you sudo pip install something that has a conflicting dependency, it bypasses OS packaging, and *boom*. I find tools like pipx go a long way to solve this, as they install a Python package and all of its dependencies in its own venv. This is great for Python apps, and (kinda) treats them like apps on platforms like Android, where all app dependencies are bundled and isolated from others. I think it would great if OS vendors did something similar to pipx for Python-based apps: bundle the app and all of its dependencies into its own venv. On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.
How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/KMRNKS... Code of Conduct: http://python.org/psf/codeofconduct/
![](https://secure.gravatar.com/avatar/e91a97baa90dd7f5a267c79a2cbb04b2.jpg?s=120&d=mm&r=g)
I love pipx and I'm glad it exists at this point because it make The main issue is that each virtualenv takes space, lots of space. I have currently 57 apps installed via pipx on my laptop, and the 57 environments take almost 1 GB already. ~ cd .local/pipx/venvs/ ~/.l/p/venvs ls abilian-tools/ concentration/ gitlabber/ pygount/ sphinx/ ansible/ cookiecutter/ httpie/ pyinfra/ tentakel/ assertize/ cruft/ isort/ pylint/ tlv/ autoflake/ cython/ jupyterlab/ pyre-check/ towncrier/ black/ dephell/ lektor/ pytype/ tox/ borgbackup/ docformatter/ md2pdf/ pyupgrade/ twine/ borgmatic/ flake8/ medikit/ radon/ virtualenv/ bpytop/ flit/ mypy/ re-ver/ virtualfish/ check-manifest/ flynt/ nox/ sailboat/ vulture/ clone-github/ gh-clone/ pdoc3/ salvo/ cloneall/ ghtop/ pdocs/ shed/ com2ann/ gitchangelog/ pybetter/ sixer/ ~/.l/p/venvs du -sh . 990M . ~/.l/p/venvs ls | wc 57 57 475 There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations. Still, I'm glad that pipx exists as it is now, and that it has been packaged on Ubuntu 20.04 and later (and probably other distros as well). Having pipx (or something similar) installed by the distro, and the distro focussed on packaging only the packages that are needed for its own sake, means that we could go past the controversies between the Python community and the Debian (or other distros) packagers community, which are based on different goals and assumption, such as this one: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e S. On Wed, Feb 24, 2021 at 2:28 AM Paul Bryan <pbryan@anode.ca> wrote:
I think it's a classic case of dependency hell.
OS packagers are rebundling Python packages as OS packages and expressing their own OS-package dependency graphs. Then, you sudo pip install something that has a conflicting dependency, it bypasses OS packaging, and *boom*.
I find tools like pipx <https://pipxproject.github.io/pipx/> go a long way to solve this, as they install a Python package and all of its dependencies in its own venv. This is great for Python apps, and (kinda) treats them like apps on platforms like Android, where all app dependencies are bundled and isolated from others.
I think it would great if OS vendors did something similar to pipx for Python-based apps: bundle the app and all of its dependencies into its own venv.
On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
I was reading a discussion thread < https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.
How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/KMRNKS... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2VYJAJHH... Code of Conduct: http://python.org/psf/codeofconduct/
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
![](https://secure.gravatar.com/avatar/33bd15feb2558d0050e863875e0f5f60.jpg?s=120&d=mm&r=g)
On 24/02/2021 11.52, Stéfane Fermigier wrote:
I love pipx and I'm glad it exists at this point because it make
The main issue is that each virtualenv takes space, lots of space.
I have currently 57 apps installed via pipx on my laptop, and the 57 environments take almost 1 GB already.
~ cd .local/pipx/venvs/ ~/.l/p/venvs ls abilian-tools/ concentration/ gitlabber/ pygount/ sphinx/ ansible/ cookiecutter/ httpie/ pyinfra/ tentakel/ assertize/ cruft/ isort/ pylint/ tlv/ autoflake/ cython/ jupyterlab/ pyre-check/ towncrier/ black/ dephell/ lektor/ pytype/ tox/ borgbackup/ docformatter/ md2pdf/ pyupgrade/ twine/ borgmatic/ flake8/ medikit/ radon/ virtualenv/ bpytop/ flit/ mypy/ re-ver/ virtualfish/ check-manifest/ flynt/ nox/ sailboat/ vulture/ clone-github/ gh-clone/ pdoc3/ salvo/ cloneall/ ghtop/ pdocs/ shed/ com2ann/ gitchangelog/ pybetter/ sixer/ ~/.l/p/venvs du -sh . 990M. ~/.l/p/venvs ls | wc 57 57 475
There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations.
There are tools like https://rdfind.pauldreik.se/rdfind.1.html that create hard links to deduplicate files. Some files systems have deduplicated baked in, too. Christian
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier <sf@fermigier.com> wrote:
There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations.
Ultimately the problem is that a general tool can't deal with conflicts (except by raising an error). If application A depends on lib==1.0 and application B depends on lib==2.0, you simply can't have a (consistent) environment that supports both A and B. But that's the rare case - 99% of the time, there are no conflicts. One env per app is a safe, but heavy handed, approach. Managing environments manually isn't exactly *hard*, but it's annoying manual work that pipx does an excellent job of automating, so it's a disk space vs admin time trade-off. As far as I know, no-one has tried to work on the more complex option of sharing things (pipx shares the copies of pip, setuptools and wheel that are needed to support pipx itself, but doesn't extend that to application dependencies). It would be a reasonable request for pipx to look at, or for a new tool, but I suspect the cost of implementing it simply outweighs the benefit ("disk space is cheap" :-)) Paul
![](https://secure.gravatar.com/avatar/e91a97baa90dd7f5a267c79a2cbb04b2.jpg?s=120&d=mm&r=g)
On Wed, Feb 24, 2021 at 12:42 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier <sf@fermigier.com> wrote:
There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations.
Ultimately the problem is that a general tool can't deal with conflicts (except by raising an error). If application A depends on lib==1.0 and application B depends on lib==2.0, you simply can't have a (consistent) environment that supports both A and B. But that's the rare case - 99% of the time, there are no conflicts. One env per app is a safe, but heavy handed, approach. Managing environments manually isn't exactly *hard*, but it's annoying manual work that pipx does an excellent job of automating, so it's a disk space vs admin time trade-off.
There are three ways to approach the question: 1) Fully isolated envs. The safest option but uses the most space. 2) Try to minimise the number of dependencies installed by interpreting the requirements specification in the looser way possible. This is both algorithmically hard (see https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and risky, as you've noted. 3) But the best way IMHO is to compute dependencies for each virtualenv independently from the others, but still share the packages, using some indirection mechanisms (hard links, symlinks or some Python-specific constructs) when the versions match exactly. The 3rd solution is probably the best of the 3, but the sharing mechanism still needs to be specified (and, if needed, implemented) properly. I've tried Christian's suggestions of using rdfind on my pipx installation, and it claims to reduce the footprint by 30% (nice, but less than I expected. This would however scale better with the number of installed packages). I'm not sure this would be practical in reality, OTOH, because I think there is a serious risk of breakage each time I would upgrade one of the packages (via 'pipx upgrade-all' for instance). So IMHO the best way to implement solution 3 would be by using some variant of the approach popularized by Nix (repository of immutable packages + links to each virtualenv). S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
![](https://secure.gravatar.com/avatar/e91a97baa90dd7f5a267c79a2cbb04b2.jpg?s=120&d=mm&r=g)
On Wed, Feb 24, 2021 at 1:47 PM Stéfane Fermigier <sf@fermigier.com> wrote:
So IMHO the best way to implement solution 3 would be by using some variant of the approach popularized by Nix (repository of immutable packages + links to each virtualenv).
Another benefit of this kind of approach, besides sparing disk space, could be similar improvements in terms of installation time (and even bigger improvements when reinstalling a package, which happens all the time when developing). S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
![](https://secure.gravatar.com/avatar/392880c03b3481024eca1b762ffe3ff2.jpg?s=120&d=mm&r=g)
On 2021-02-24 02:52, Stéfane Fermigier wrote:
I have currently 57 apps installed via pipx on my laptop, and the 57 environments take almost 1 GB already.
I never understood the fear around version conflicts. Perhaps it has to do with the decline of sys-admin skills over the years? So, the strategy above feels like swatting a fly with a sledgehammer to me. Same as with a venv for every app. :-) Rather than installing every package this way, why not wait until a conflict actually occurs? Personally, I rarely have such conflicts, maybe every few years or so. When it happens, I fix them by uninstalling the offender and putting the more difficult one into the venv or pipx. Right now I only have one, a giant app from work that uses pipenv, and it's fine. Now what about sudo and all that? Well, I used it in the old days because that's what the instructions said. But, to be honest, it never made any sense. I haven't shared a computer in decades, and when we did we used NFS for common tools, so it never saved any disk space. Pip (and easy_install?) dragged their feet for years to properly support user installs (should have been default) but once they did I didn't look back. I dump almost all packages to user, which gets cleaned up every other year when the distro moves to the next version. The strategy has been working well for a long time. So, --user works at the low end, and containers for the high end. Honestly, I don't have much of a use case any longer for venvs. -Mike
![](https://secure.gravatar.com/avatar/ded73f8e2dc9541eca376ca4f759de4a.jpg?s=120&d=mm&r=g)
On Wed, Feb 24, 2021 at 8:50 PM Mike Miller <python-dev@mgmiller.net> wrote:
I never understood the fear around version conflicts.
With binary extension modules, version conflicts lead to (at best) runtime segfault and (at worst) subtle *data* bugs that return incorrect results. There are also deeper concerns around security and reproducibility.
Perhaps it has to do with the decline of sys-admin skills over the years?
Many millions of users of new Python users show up every year, using the language and its powerful ecosystem for data analytics and scientific computing, and they have no hope of having sys-admin skills. -Peter
![](https://secure.gravatar.com/avatar/63ca18e130d527d0741f1da54bb129a7.jpg?s=120&d=mm&r=g)
The challenge with version conflicts is often less that you need to go update the constraints (which has little to do with sysadmin'ing, TBH) and more that you have insufficient *integration tests* and you're relying upon something else running the per-package tears. On Thu, Feb 25, 2021, 00:10 Peter Wang <pwang@anaconda.com> wrote:
On Wed, Feb 24, 2021 at 8:50 PM Mike Miller <python-dev@mgmiller.net> wrote:
I never understood the fear around version conflicts.
With binary extension modules, version conflicts lead to (at best) runtime segfault and (at worst) subtle *data* bugs that return incorrect results. There are also deeper concerns around security and reproducibility.
Perhaps it has to do with the decline of sys-admin skills over the years?
Many millions of users of new Python users show up every year, using the language and its powerful ecosystem for data analytics and scientific computing, and they have no hope of having sys-admin skills.
-Peter
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TYZ6QT4B... Code of Conduct: http://python.org/psf/codeofconduct/
![](https://secure.gravatar.com/avatar/43bc83462d0c9d7f65a783c00f9189ad.jpg?s=120&d=mm&r=g)
On Thu, Feb 25, 2021 at 5:35 AM Wes Turner <wes.turner@gmail.com> wrote:
The challenge with version conflicts is often less that you need to go update the constraints (which has little to do with sysadmin'ing, TBH) and more that you have insufficient *integration tests* and you're relying upon something else running the per-package tears.
Sometimes, auto-correct really does understand! Your point is right on target, and really can't be understated. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "There is nothing more uncommon than common sense." --Frank Lloyd Wright
![](https://secure.gravatar.com/avatar/63ca18e130d527d0741f1da54bb129a7.jpg?s=120&d=mm&r=g)
One aspect of conda packaging that's worth adopting is the 'test' block of the meta.yaml which contains commands necessary to *test* the built package. You can add a pytest command to setup.py, but it's not as easy as: https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml... : ```yaml test: downstreams: - cython - setuptools requires: - ripgrep - cmake - ninja - {{ compiler('c') }} # Tried to use enable_language(C) to avoid needing this. It does not work. - {{ compiler('cxx') }} files: - tests/distutils/* - tests/cmake/* - tests/cython/* - tests/prefix-replacement/* commands: - echo on # [win] - set # [win] - python -V - python3 -V # [not win] - 2to3 -h - pydoc -h - python3-config --help # [not win] - set "PIP_NO_BUILD_ISOLATION=False" # [win] - set "PIP_NO_DEPENDENCIES=True" # [win] - set "PIP_IGNORE_INSTALLED=True" # [win] - set "PIP_NO_INDEX=True" # [win] - set "PIP_CACHE_DIR=%CONDA_PREFIX%/pip_cache" # [win] - set "TEMP=%CONDA_PREFIX%/tmp" # [win] - mkdir "%TEMP%" # [win] - python -Im ensurepip --upgrade --default-pip # [win] # tzdata/zoneinfo test that will need the tzdata package to pass - python -c "from zoneinfo import ZoneInfo; from datetime import datetime; dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo('America/Los_Angeles')); print(dt.tzname())" - python -m venv test-venv - python -c "import sysconfig; print(sysconfig.get_config_var('CC'))" # [not win] - _CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu python -c "import sysconfig; print(sysconfig.get_config_var('CC'))" # [linux64] # check for unreplaced @ symbols in sysconfig files - for f in ${CONDA_PREFIX}/lib/python*/_sysconfig*.py; do echo "Checking $f:"; if [[ `rg @ $f` ]]; then echo "FAILED ON $f"; cat $f; exit 1; fi; done # [linux64 or osx] - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.a # [unix] - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.nolto.a # [unix] # https://github.com/conda-forge/python-feedstock/issues/384 - if exist %PREFIX%\\Scripts\\pydoc exit 1 # [win] - if exist %PREFIX%\\Scripts\\idle exit 1 # [win] - if exist %PREFIX%\\Scripts\\2to3 exit 1 # [win] - if not exist %PREFIX%\\Scripts\\pydoc-script.py exit 1 # [win] - if not exist %PREFIX%\\Scripts\\idle-script.py exit 1 # [win] - if not exist %PREFIX%\\Scripts\\2to3-script.py exit 1 # [win] - if not exist %PREFIX%\\Scripts\\idle.exe exit 1 # [win] - if not exist %PREFIX%\\Scripts\\2to3.exe exit 1 # [win] - if not exist %PREFIX%\\Scripts\\pydoc.exe exit 1 # [win] - pushd tests - pushd distutils - python setup.py install -v -v - python -c "import foobar" - popd - pushd prefix-replacement # [unix] - bash build-and-test.sh # [unix] - popd # [unix] - pushd cmake - cmake -GNinja -DPY_VER={{ version }} # --trace --debug-output --debug-trycompile . - popd - popd - test ! -f default.profraw # [osx] ``` In which package you then store integration tests for your whole application and it's integrated dependencies is then up to you. Unfortunately, we often don't include tests/ in our packages, so it's literally not possible to run the tests in production even if you have pytest_requires installed. Perhaps distros would do well to implement support for integration and per-package tests. On Thu, Feb 25, 2021, 12:55 Fred Drake <fred@fdrake.net> wrote:
On Thu, Feb 25, 2021 at 5:35 AM Wes Turner <wes.turner@gmail.com> wrote:
The challenge with version conflicts is often less that you need to go update the constraints (which has little to do with sysadmin'ing, TBH) and more that you have insufficient *integration tests* and you're relying upon something else running the per-package tears.
Sometimes, auto-correct really does understand!
Your point is right on target, and really can't be understated.
-Fred
-- Fred L. Drake, Jr. <fred at fdrake.net> "There is nothing more uncommon than common sense." --Frank Lloyd Wright
![](https://secure.gravatar.com/avatar/392880c03b3481024eca1b762ffe3ff2.jpg?s=120&d=mm&r=g)
On 2021-02-24 21:08, Peter Wang wrote:
With binary extension modules, version conflicts lead to (at best) runtime segfault and (at worst) subtle *data* bugs that return incorrect results. There are also deeper concerns around security and reproducibility.
If your app has requirements, by all means use an isolated environment. Wasn't recommending against it, when needed. Rather, that it is not a requirement for *every single script,* or even most. As we've seen some folks have been scared into a heavy-handed solution.
Perhaps it has to do with the decline of sys-admin skills over the years?
Many millions of users of new Python users show up every year, using the language and its powerful ecosystem for data analytics and scientific computing, and they have no hope of having sys-admin skills.
"sys-admin" is a bit of an overstatement in my phrasing. The core is that you need to understand how a PATH works and be able to run pip and cd into folders and perhaps run rm/del, etc. Basic command-line skills? In any case, if you can't do these, virtual envs are not a full solution either. -Mike
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Mike Miller writes:
"sys-admin" is a bit of an overstatement in my phrasing. The core is that you need to understand how a PATH works and be able to run pip and cd into folders and perhaps run rm/del, etc. Basic command-line skills?
That's what I would mean by basic sys-admin skills. And *surprise!* my students don't have them, and don't need them ... until they start using Python. They don't even understand installing packages in conda. If I had my druthers, we wouldn't have classes in Excel and Powerpoint, and we wouldn't have classes in SPSS and SAS, and we wouldn't even have classes in Python -- we'd have classes in envars, registry, and shell scripts/bat files.
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Fri, Feb 26, 2021 at 11:41:56AM +0900, Stephen J. Turnbull wrote:
Mike Miller writes:
"sys-admin" is a bit of an overstatement in my phrasing. The core is that you need to understand how a PATH works and be able to run pip and cd into folders and perhaps run rm/del, etc. Basic command-line skills?
That's what I would mean by basic sys-admin skills. And *surprise!* my students don't have them, and don't need them ... until they start using Python.
Is it *only* Python though? Wouldn't that be necessary if they were learning Perl, Java, C, Ruby etc? -- Steve
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Fri, 26 Feb 2021 at 09:07, Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Feb 26, 2021 at 11:41:56AM +0900, Stephen J. Turnbull wrote:
Mike Miller writes:
"sys-admin" is a bit of an overstatement in my phrasing. The core is that you need to understand how a PATH works and be able to run pip and cd into folders and perhaps run rm/del, etc. Basic command-line skills?
That's what I would mean by basic sys-admin skills. And *surprise!* my students don't have them, and don't need them ... until they start using Python.
Is it *only* Python though? Wouldn't that be necessary if they were learning Perl, Java, C, Ruby etc?
It is possible to teach students to do most things in Python without using the command line. For example you can tell them all to install anaconda and then use spyder as their editor. Or you can get them to install and use vscode or pycharm or some other IDE with built in virtualenv and git integration etc. Where Python is awkward is at the point where you actually *want* to teach students to use the command line. The problem is that Python is not set up for command line use out of the box in a consistent way across different platforms. It would be nice to say: now open a terminal, type "python", and hit the enter key and you'll see the Python prompt >>> but it's just not that simple so the instructions end up being like: """ It might be possible to run python from the terminal by typing "python". However that might not work or it might run the wrong version/installation of Python. It might be that you haven't added Python to your PATH environment variable or maybe you have depending on which boxes you ticked when installing. Possibly you need to change your PATH environment variable or possibly you should run Python as "python3" or "py" instead but I have no way of saying without knowing what OS you are on, where you installed Python from, what options you selected in the installer, whether you have any other Python installations etc. If you are on Windows and you get "permission denied" then that's possibly because the MS-provided python.exe stub is on your PATH so you first need to disable that in "app execution aliases". Possibly you can't run Python from the normal terminal but you can open a special terminal for Python that does have the PATH set up such as the Anaconda Prompt. Okay next lesson is how to use "pip" which is really easy: just type "pip install X" but maybe you should actually type "pip3" or "python -m pip" or "python3 -m pip" or "py -m pip" or actually you if you installed Anaconda you might need to use "conda install X". """ When someone learns to use the command line for the first time it's really quite helpful if you can tell them exactly what commands to type. Knowing how to configure PATH and understanding what it does and understanding about OS differences etc is potentially useful in the long run but is a major distraction at the beginning. There have been efforts to improve the situation but in some ways they make it worse. For example the py launcher supposedly fixes some of these problems on Windows but the fact that there is no analogue of it on any non-Windows platform means that any instructions involving it immediately need to split according to OS. Likewise the new python.exe stub that ships with Windows is supposed to fix the Python PATH problem but it actually *prevents* python from working in the terminal if you have installed Python some other way (even if you have added it to PATH): https://stackoverflow.com/questions/56974927/permission-denied-trying-to-run... The problems aren't only on Windows because you also have "python" vs "python3" on OSX, Linux etc. On OSX you can also have conflicts between the system Python and any user installed Python. On OSX some installers set up the PATH for you by automagically editing .bash_profile which can be helpful but can also be confusing. After some time people don't actually know how to run the different versions of Python that they installed through homebrew, from python.org or from Anaconda etc and their .bash_profile is a mess. The question of exactly what you should type to run python, pip, etc applies on Linux although I find it less problematic just because Linux users typically know how to use the terminal already. For me as a user of Python I don't usually notice these problems because I use some sort of virtual environment for everything so I can always type "python" or "pip" and there are no conflicts between different installations. When a student can't even get "python" to work in the terminal though it's too much to ask them to start setting up virtual environments. -- Oscar
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Fri, 26 Feb 2021 at 13:31, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Where Python is awkward is at the point where you actually *want* to teach students to use the command line. The problem is that Python is not set up for command line use out of the box in a consistent way across different platforms.
My experience with Java is that it's at least as bad. (I'd say "worse", but that might just be because I'm more familiar with Python than with Java). For C, do I type cc, cl, gcc, clang, ...? Do I need to start a "Visual Studio command prompt"? What if I prefer using powershell? For Ruby and Perl, I have to install them, and assuming that's easy, I've no idea what to do then. I'm not sure what the point is here though. If people without command line skills want to use the command line, they'll struggle? Well, yes, that seems likely. And I have no idea how this relates to virtual environments vs the system Python. Paul So yes, Python's experience sucks, but so do most languages.
![](https://secure.gravatar.com/avatar/d91ce240d2445584e295b5406d12df70.jpg?s=120&d=mm&r=g)
I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point. The problem is made worse because it often hits at the beginning instead of 7 lessons in, when they would have some reason to think it is worth struggling through and will eventually work.
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett <jimjjewett@gmail.com> wrote:
I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.
Thanks Jim, that is the situation and yes they are not CS students. The other point though is that it doesn't need to be like this. If the issue was just installing Python and then setting up your PATH then that's manageable. The problem is that even after doing those things there isn't a "one way" to invoke Python from the command line. All possible invocations (python, python3, py, ...) will fail for some users. That's a problem for beginners but it's also a problem in any situation where you want to write a command line that should run on multiple platforms (e.g. in a Makefile or some other kind of script). I see that the official Python tutorial now suggests typing "python3.9" [1]. Is that what is recommended now? Obviously that would fail for someone who had installed 3.8. It would be great if Python could converge around a single way for users to invoke the Python that they have installed and would want to use. Python 2.x is beginning to fade away so python vs python3 makes less sense now but maybe standardising on py for all platforms and installs would be better (since py also has features for selecting different versions and is not currently used for any "system python"). [1]: https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter Oscar
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Sat, Feb 27, 2021 at 01:04:08AM +0000, Oscar Benjamin wrote:
On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett <jimjjewett@gmail.com> wrote:
I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.
Thanks Jim, that is the situation and yes they are not CS students.
Why is that relevant? I'm not an automotive engineer, I haven't studied for an Automotive Engineering Technology (AET) bachelor degree, but I still need to know how to change a flat tire, fill the petrol tank with fuel, keep the windscreen wiper water tank filled, put snow chains on if I live in a region with snow, etc. Or pay somebody else to do it. Python is not Scratch. It doesn't, and shouldn't, aim to be a self-contained environment that isolates the user from their own computer. [Aside: the Scratch interpreter will actually refuse to run if it thinks your code or documentation includes "bad words": https://i.redd.it/nyqra8lmbei21.png https://scratch.mit.edu/discuss/topic/32001/ https://scratch.mit.edu/discuss/topic/122410/ If this doesn't make you want to take off and nuke the entire planet from orbit, then I don't know what's wrong with you. End of rant.] It doesn't matter whether you are using Windows, Mac OS X or Linux, whether your preferred UI is the command line or point-and-click, there are some basic fundemental skills that every amateur or professional programmer needs to know, such as: - how to edit files and save them - which file am I actually running? - which interpreter am I actually running? - how do I tell the computer to use a different interpreter? These should be basic pre-requisites before learning to code, like being able to read and write. [...]
I see that the official Python tutorial now suggests typing "python3.9" [1]. Is that what is recommended now? Obviously that would fail for someone who had installed 3.8.
It isn't elitist to expect that, at the barest minimum, any student of programming should be able to adjust the command python3.9 hello.py to python3.8 myscript.py as required. If you can't debug python3.9: command not found or can't open file '/home/steve/hello.py': [Errno 2] No such file or directory then you're probably going to have a miserable time dealing with *hard* bugs. Other more subtle errors may require a bit more knowledge, but that's okay. We don't need to make programming a knowledge-free activity.
The other point though is that it doesn't need to be like this. If the issue was just installing Python and then setting up your PATH then that's manageable. The problem is that even after doing those things there isn't a "one way" to invoke Python from the command line. All possible invocations (python, python3, py, ...) will fail for some users. That's a problem for beginners but it's also a problem in any situation where you want to write a command line that should run on multiple platforms (e.g. in a Makefile or some other kind of script).
Writing code to run on any platform is a hard problem that requires complex solutions. If you need to write a makefile that will run on a million different flavours of Unix, including rare, old and weird ones, then you have to expect that's a hard problem and you're probably going to need to spend some effort getting it to work. Not every problem is capable of being solved by a beginner with no knowledge, and we don't have to bust our guts to make it an easy problem. It's okay to delegate some problems to someone else, such as make experts.
It would be great if Python could converge around a single way for users to invoke the Python that they have installed ... py also has features for selecting different versions
I don't think that "py --someoption ..." is any better than the existing solutions I already have for selecting different versions. Maybe on Windows? I don't know how it works there. -- Steve
![](https://secure.gravatar.com/avatar/d91ce240d2445584e295b5406d12df70.jpg?s=120&d=mm&r=g)
[They are not CS students]
Why is that relevant?
Because for many users, python is NOT a programming language; it is an application like any other. It happens to be very powerful and flexible, but the point isn't to program; it is to produce better reports. If the hurdle to get started (either initially, or after a while away) is too high, then it is not a useful tool. It is OK to have all sorts of power-user knobs, but the default instructions should just work ... they shouldn't have to say "ensure you have proper settings for these 6 things you don't care about or understand", because the default install should just choose reasonable default settings and take care of them.
there are some basic fundemental skills that every amateur or professional programmer needs to know, such as:
My point is that plenty of python users have no intention of being a programmer.
how to edit files and save them (fair ... but plenty of programs don't even *let* you do this with an external editor; there is no reason the _default_ should force people to pick and configure an external editor. IDLE tends to do OK here.)
which file am I actually running? which interpreter am I actually running? how do I tell the computer to use a different interpreter?
If you need to care about any of these, then the environment is fighting you -- and the application probably stinks. Programmers have to deal with it because of bootstrapping, but there is no reason that we should assume all python users need that flexibility or want that responsibility.
Writing code to run on any platform is a hard problem that requires complex solutions.
Thus the preference for solving it once in the library/official documentation/reference installer. Experts can build on it or opt out, but non-programmers shouldn't have to worry about cross-platform issues just to use python. -jJ
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Jim J. Jewett writes:
which file am I actually running? which interpreter am I actually running? how do I tell the computer to use a different interpreter?
If you need to care about any of these, then the environment is fighting you -- and the application probably stinks. Programmers have to deal with it because of bootstrapping, but there is no reason that we should assume all python users need that flexibility or want that responsibility.
Be careful! I agree about need for flexibility and desire for responsibility, but in fact these problems present themselves frequently when helping people with their work. "Which file am I actually [edit]ing?" is one that comes up in just about every meeting: documents are undated (or dated with the deadline date), people are looking at different versions, and not infrequently the presenter themself has a Documents/Downloads confusion and gets it wrong. Then "where is the one I want to be editing?" requires an oracle. :-) "Which program am I running?" doesn't come up that often, but it does come up when helping people with "doing something", especially trying to find menu items across major versions. Interestingly enough, "how do I tell the computer to use a different interpreter?" doesn't come up for me outside of Python (and even there rarely) -- everybody who consults me knows how to use "Open With". So I agree with the basic principle that programs should Just Work by default, but it's not always possible. Eg, there's good reason why Downloads and Documents both exist. So it helps to have users know something about implementations and sysadminning stuff, especially when remote troubleshooting. The question for Python is given the history and the plethora of ways to invoke Python and find packages and programs, can we do better with a minimum of backward incompatibility? Humans are generally better at learning this stuff than bags of bits on spinning platters are!
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Sun, 28 Feb 2021 at 07:04, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Jim J. Jewett writes:
which file am I actually running? which interpreter am I actually running? how do I tell the computer to use a different interpreter?
If you need to care about any of these, then the environment is fighting you -- and the application probably stinks. Programmers have to deal with it because of bootstrapping, but there is no reason that we should assume all python users need that flexibility or want that responsibility.
<snip>
So I agree with the basic principle that programs should Just Work by default, but it's not always possible. Eg, there's good reason why Downloads and Documents both exist. So it helps to have users know something about implementations and sysadminning stuff, especially when remote troubleshooting.
The question for Python is given the history and the plethora of ways to invoke Python and find packages and programs, can we do better with a minimum of backward incompatibility? Humans are generally better at learning this stuff than bags of bits on spinning platters are!
I think that the py launcher is actually a good solution to many of these problems. It can also be improved but the main reason it doesn't help right now is just the fact that it's only for Windows and also isn't always installed on Windows. When the launcher is installed though then: - It is on PATH - It can list the Python installations with "py -0p" - It can be used to select a particular Python version/installation to use from the command line e.g. "py -3.8 myscript.py" - It is possible to configure a default version (although I think you have to do it with an environment variable) There are some ways that the launcher situation could be improved though: - I think that the launcher is only installed in an all users install. There would need to be a way to do it in user-space as well (even if it means the user has to configure PATH). - Listing installations with "py -0p" is somewhat cryptic - It would be better if you could use the launcher itself to set the default Python e.g. "py --set-default-python=3.8" - The launcher isn't installed by Anaconda On the last point I think that although Anaconda doesn't install the launcher you can use the launcher to run the python from the Anaconda installation. Ideally Anaconda would install the launcher but it could also be possible to install it separately. If py was provided on OSX, Linux etc along with the very clear guidance that it is supposed to be for "users" rather than the "system" then maybe that could be a better situation. It would be backwards compatible since the launcher already exists on Windows and "py" hasn't been used for anything on non-Windows platforms (AFAIK). -- Oscar
![](https://secure.gravatar.com/avatar/460e6e299e385fd46df156c85a1dd55f.jpg?s=120&d=mm&r=g)
On 2/28/21, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
- It is possible to configure a default version (although I think you have to do it with an environment variable)
The py launcher in Windows supports a "py.ini" file beside the executable and in %LocalAppData%. The equivalent of the PY_PYTHON, PY_PYTHON2, and PY_PYTHON3 environment variables can be set in the "[defaults]" section as "python", "python2", and "python3" settings. The ini file also supports a "[commands]" section to define additional virtual commands for shebangs. Regular filepaths are also supported in shebangs -- e.g. #!"path\to\any\program.exe".
- I think that the launcher is only installed in an all users install.
It defaults to an all-users install, but it can also be installed for just the current user in "%LocalAppData%\Programs\Python\Launcher". In this case, the installation directory always gets added to PATH.
- Listing installations with "py -0p" is somewhat cryptic
There's also the long-form options `--list` and `--list-paths`.
- It would be better if you could use the launcher itself to set the default Python e.g. "py --set-default-python=3.8"
It's pretty simply to run `set PY_PYTHON=3.8`, and persist the value in the registry with `setx.exe PY_PYTHON 3.8`. (But don't use setx.exe naively to set PATH.)
On the last point I think that although Anaconda doesn't install the launcher you can use the launcher to run the python from the Anaconda installation.
I don't use Anaconda, but I don't think that's supposed to be the case according to PEP 514. The launcher only looks for PSF development distributions in the "PythonCore" registry key.
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Mon, 1 Mar 2021 at 00:13, Eryk Sun <eryksun@gmail.com> wrote:
On 2/28/21, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
- It is possible to configure a default version (although I think you have to do it with an environment variable)
The py launcher in Windows supports a "py.ini" file beside the executable and in %LocalAppData%. The equivalent of the PY_PYTHON, PY_PYTHON2, and PY_PYTHON3 environment variables can be set in the "[defaults]" section as "python", "python2", and "python3" settings.
The ini file also supports a "[commands]" section to define additional virtual commands for shebangs. Regular filepaths are also supported in shebangs -- e.g. #!"path\to\any\program.exe".
Yes, I forgot about that. I still think it would be better if you could do it with the py command...
- I think that the launcher is only installed in an all users install.
It defaults to an all-users install, but it can also be installed for just the current user in "%LocalAppData%\Programs\Python\Launcher". In this case, the installation directory always gets added to PATH.
Oh, okay. So does that mean that it's always on PATH unless the user *explicitly unticks* the "install the launcher" box for both single user and all user installs?
- It would be better if you could use the launcher itself to set the default Python e.g. "py --set-default-python=3.8"
It's pretty simply to run `set PY_PYTHON=3.8`, and persist the value in the registry with `setx.exe PY_PYTHON 3.8`. (But don't use setx.exe naively to set PATH.)
Those commands aren't portable to any other OS though. The advantage of having it done through py is that it can (hopefully) be made consistent across platforms.
On the last point I think that although Anaconda doesn't install the launcher you can use the launcher to run the python from the Anaconda installation.
I don't use Anaconda, but I don't think that's supposed to be the case according to PEP 514. The launcher only looks for PSF development distributions in the "PythonCore" registry key.
Maybe I misunderstood the situation. If py can only be used for PSF binaries then that wouldn't work. I guess that can be changed though... -- Oscar
![](https://secure.gravatar.com/avatar/460e6e299e385fd46df156c85a1dd55f.jpg?s=120&d=mm&r=g)
On 2/28/21, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Oh, okay. So does that mean that it's always on PATH unless the user *explicitly unticks* the "install the launcher" box for both single user and all user installs?
If the launcher gets installed, it will be available in PATH. IIRC, the installer only allows installing the launcher for the current user if it is not installed already for all users. Thus only one version should ever exist in PATH, which is set in either the user or system "PATH" value in the registry, but not both.
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Sat, 27 Feb 2021 at 01:08, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett <jimjjewett@gmail.com> wrote:
I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.
Thanks Jim, that is the situation and yes they are not CS students.
I've been in this situation with people who are professional developers (but in environments where "development" is done in a self-contained application IDE). I completely understand and sympathise with the idea that suddenly having to learn command line skills when all you'd signed up for was to learn Python is both daunting and off-putting.
The other point though is that it doesn't need to be like this. If the issue was just installing Python and then setting up your PATH then that's manageable. The problem is that even after doing those things there isn't a "one way" to invoke Python from the command line. All possible invocations (python, python3, py, ...) will fail for some users. That's a problem for beginners but it's also a problem in any situation where you want to write a command line that should run on multiple platforms (e.g. in a Makefile or some other kind of script).
This is 100% true. (At least the part where you say that "it's a problem". I'm not sure if "it doesn't need to be like this" is true - can you point to a language where the command line "just works" the way you suggest?)
I see that the official Python tutorial now suggests typing "python3.9" [1]. Is that what is recommended now? Obviously that would fail for someone who had installed 3.8.
It would be great if Python could converge around a single way for users to invoke the Python that they have installed and would want to use. Python 2.x is beginning to fade away so python vs python3 makes less sense now but maybe standardising on py for all platforms and installs would be better (since py also has features for selecting different versions and is not currently used for any "system python").
It *would* be great. I don't think there's anyone who disagrees with this point. But just thinking "it would be great" isn't of much practical use. People have been trying to fix this issue for many years (quite apart from all of the core Python debates, it's an endless, frustrating topic in the packaging community - I can't tell you how many times we've tried to come up with a good way to tell people how to invoke pip). In my view, for people learning Python in a context where command line skills are *not* the norm, it would be ideal to focus on one (or more) "integrated environment", in much the same way as no-one learns command-line R, they all use R Studio. Unfortunately, the Python community isn't good at getting behind a single solution like this, so we have IDLE, Jupyter, Spyder, and probably a host of others. The command is *hard*. People used to it tend to forget that. I know people who are good developers (in an IDE) who struggle with how to specify options to commands, and how to quote arguments. (Not surprising, as both of those depend on what OS and what shell you're using, as well as being entirely learned information - there's no "intuitive" reason why options start with a dash, or you quote with double quotes not single quotes in Windows). I don't think we should be pointing beginners at the command line *at all*. But as a community, we don't really have any better option. (Individual trainers or organisations can make their own choices, but that's not easy in the absence of a community consensus). Paul PS Having a standard "integrated environment" that people learn in has its own problems, of course, as we risk training a new generation of Python users who don't know how to use their Python code outside of that IDE, and who therefore can't work with git or automated tests, etc, etc... I'm looking at Java and the "if my IDE doesn't do it, I'm lost" mindset here. I don't know if R has fallen into a similar trap with R Studio.
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Sat, 27 Feb 2021 at 11:04, Paul Moore <p.f.moore@gmail.com> wrote:
On Sat, 27 Feb 2021 at 01:08, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
The other point though is that it doesn't need to be like this. If the issue was just installing Python and then setting up your PATH then that's manageable. The problem is that even after doing those things there isn't a "one way" to invoke Python from the command line. All possible invocations (python, python3, py, ...) will fail for some users. That's a problem for beginners but it's also a problem in any situation where you want to write a command line that should run on multiple platforms (e.g. in a Makefile or some other kind of script).
This is 100% true. (At least the part where you say that "it's a problem". I'm not sure if "it doesn't need to be like this" is true - can you point to a language where the command line "just works" the way you suggest?)
Python from 15 years ago :) Maybe I'm misremembering but when I first started with Python there didn't seem to be any ambiguity about what to type in the terminal. The first split as I remember it was on Linux when python3 came along but python kept referring to 2.x on (most) Linux. At the same time python3 was not added to Windows and instead python became 3.x. Then there was the py launcher for Windows. Perhaps this problem always existed on OSX although I didn't use it at all at the time. More recently I've been exploring Julia. You could compare these pages: https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter https://docs.julialang.org/en/v1/manual/getting-started/ When installing julia you select the installer for your OS and there are some instructions for setting up PATH (although I didn't need to do that manually on OSX). After setup the instruction is that you type "julia" in the terminal and then Ctrl-D to exit. There are no caveats or mentions of different operating systems or words like "if" and "usually". The setup process is platform specific but then the usage instructions are not. -- Oscar
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Sat, Feb 27, 2021 at 12:56:17PM +0000, Oscar Benjamin wrote:
On Sat, 27 Feb 2021 at 11:04, Paul Moore <p.f.moore@gmail.com> wrote:
This is 100% true. (At least the part where you say that "it's a problem". I'm not sure if "it doesn't need to be like this" is true - can you point to a language where the command line "just works" the way you suggest?)
Python from 15 years ago :)
Fifteen years ago, if I remember correctly, I had Python 1.5, 2.4 and 2.5 installed. (I may have some of the versions mixed up.) When I typed `python` at the command line, the OS read the faint electrical currents at my finger tips through the keyboard, extrapolated the state of my brain, predicted the interpreter I wanted, and unfailingly guessed the right one. Nah just kidding. I'm pretty sure it ran 2.4, always.
When installing julia you select the installer for your OS and there are some instructions for setting up PATH (although I didn't need to do that manually on OSX). After setup the instruction is that you type "julia" in the terminal and then Ctrl-D to exit. There are no caveats or mentions of different operating systems or words like "if" and "usually". The setup process is platform specific but then the usage instructions are not.
So what happens when I have two different Julia versions installed? -- Steve
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Sat, 27 Feb 2021 at 13:03, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
More recently I've been exploring Julia. You could compare these pages:
https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter https://docs.julialang.org/en/v1/manual/getting-started/
When installing julia you select the installer for your OS and there are some instructions for setting up PATH (although I didn't need to do that manually on OSX). After setup the instruction is that you type "julia" in the terminal and then Ctrl-D to exit. There are no caveats or mentions of different operating systems or words like "if" and "usually". The setup process is platform specific but then the usage instructions are not.
Sorry, I couldn't resist replying to this. I went to the Julia page. That linked me to the download page. Which offered me multiple versions. Which should I take? Stable, LTS or "upcoming"? As a beginner without the experience I have hitting stuff like this, how would I know? Let's assume "stable". OK, download and run an installer is easy enough for me on Windows, but let's say I'm doing this on Linux, I get the choice of "Generic Linux on x86" or "Generic Linux on ARM". Huh? I'm on Ubuntu, not "Generic", so I'm confused. Then I have "64-bit (GPG)" or "64-bit (musl) (GPG)" Or AArch if I chose "ARM". How the heck would I know. Any of those get me a .tar.gz file, which I don't know how to use... Going to the Linux help, I get told to use wget, not the file I just downloaded. And then I unpack it, and now I'm into setting PATH to /tmp/julia-something/bin (because I did this in /tmp, as I have no idea where I want to put this long term). OK, my linux thought experiment isn't going well. Let's go back to Windows, where I was lucky enough to pick "installer" rather than "portable", so I avoided a world of pain about where to put the package, just like I hit on Linux. The installer puts Julia in my personal apps directory, just like Python does. I'm going to do something "advanced" here ;-) and deselect "desktop shortcut" because I hate clutter on my desktop. OK, it's complete, let's select "run julia". That got me a prompt. Cool. Start menu item does the same. Also cool. BUT... The python installer on windows is *exactly the same*. Installs to user's area, start menu entry that runs the REPL. So that's no worse. Suppose I go to a command prompt on my own and try to run Julia. Open a terminal, "julia". No good, I need to set PATH. Hmm, wasn't there something in the help on the Julia website about that? Off we go to check. Weren't we here before when we were using Python? And the instructions on the Julia website are about as complex as I'd expect, and much the same as for Python. So still no worse for Python. But the Python installer has "add Python to PATH" that does this stuff for you (not selected by default, because of reasons that you may or may not agree with, but at least it's *there*). And there's the py launcher, but let's ignore that, as it works round some complexity at the cost of some different complexity, so it's a distraction at this stage. Honestly, I think Julia is overall slightly *worse* than Python (at least on Windows). But it's almost identical in practice. The one thing that *is* substantially worse for Python, is the circumlocutions needed in the documentation to say how to run Python. But that's 100% down to us not being willing to say "just type the command python". And the reason for *that* is mostly historical, related to the Python 3 transition and what happened on Linux over the python/python2/python3 commands, and to a lesser extent the introduction of the launcher on Windows ("lesser", because using the launcher on the command line wasn't recommended until some time after its introduction, at which point the python2/python3 split was established)... Paul
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Paul Moore writes:
The one thing that *is* substantially worse for Python, is the circumlocutions needed in the documentation to say how to run Python. But that's 100% down to us not being willing to say "just type the command python". And the reason for *that* is mostly historical, related to the Python 3 transition
I don't think that's entirely true, and the roots are much earlier than Python 3. We had the same problem with Linux distributions (especially Red Hat) and version 1 vs. version 2. This was crucial for those of us handling multibyte languages: getting Unicode Inside, even in the non-str form of Python 2, really changed things. But Python 2.0 was a .0 release. For a lot of developers dealing with unibyte languages, 1.5.2 was the workhorse until about Python 2.2.
and what happened on Linux over the python/python2/python3 commands, and to a lesser extent the introduction of the launcher on Windows
I don't know about Windows. It might have been possible to arrange that just "python" was the right way to do things, and have "-2" and "-3" flags for specifying compatibility. But it just wasn't possible on *nix systems because /usr/bin/python was typically used for system software. They shouldn't have done that (they should have kept in it in an OS-specific place and used full shebangs), but that's not how it went .... I think those are important things to keep in mind when comparing with a very young language like Julia (Julia didn't make *any* backward compatibility promises until 1.0 in August 2018). A lot of Python's problems occur *because* of its multi-use potential and *because* of its wide adoption in practice.
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Executive summary: This thread is about a "seven blind men and an elephant" problem. Jim J. Jewett writes:
I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.
Not sure if "his" refers to me, but I don't know what they're going to need to use. C/C++, maybe not, but there are a lot of Perl and especially Java shops in Tokyo still. The largest group of my students end up as what the Japanese call "SE" ("systems engineers"), by which they mean a sales/requirements gathering/training/customer relations role. Many of the rest end up in tech-adjacent roles. I think all of them would benefit from a little command-line experience, and a better understanding of how a group of programs cooperates to become a system. As their teacher, I certainly would!
The problem is made worse because it often hits at the beginning instead of 7 lessons in, when they would have some reason to think it is worth struggling through and will eventually work.
I think we're on the same page, but "7 lessons in" is still early in my experience teaching Python. Languages where you have to explicitly invoke a compiler and then run the executable, maybe you run into the problem "at the beginning" (shades of Chapter 1 of _The C Programming Language_!), but not with Python. Where my students run into problems is when they *know* it was all worth it, because *now* they're ready to use Python to do whatever it is they style "real work" -- and need to install, maybe configure, and learn to use something that isn't an included battery. Even in a well-curated framework like Anaconda, "you can make a bottle adult-proof, but not child-proof". So *my* point (I don't speak for anyone else) is pessimistic. I don't think there's a one-size-fits-all solution to child-proofness. What we need is children with better vocabularies. From my point of view, what I want is to talk to students (and colleagues!) about problems that I can often solve or install a workaround in a few minutes. But when simple questions like "what distribution did you install?" gets the answer "I just downloaded the package. What's a distribution?", and the followup is, "well, where did you get it?", "I clicked on the top link in Google", which leads to "well, is it this one?" and "I can't tell...", it's very frustrating for all concerned -- and we haven't even started on the new package that isn't working. When to them, it's pure fsckin' magic, and to me, it's muscle memory, problem-solving across the gap is hard. For my purpose, cross- platform distributions like Anaconda seem to be the best solution available now and for the near future. But for many developers, they have enough problems without dealing with the idea that some other application has different version or configuration requirements for some components. I just don't see a world where any of the points of view are without merit. People with more space than apps can put them *all* in venvs, people with relatively simple "external" requirements can just use a single installation, and people with the necessary skills can mostly use the single installation and put a few "problem children" into venvs of their own. Each of those groups will want Python itself to take a different approach to the "problem". So the question in the subject is valid and should be raised once in a while, but the answer is always going to disappoint a majority. :-) Steve
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Steven D'Aprano writes:
On Fri, Feb 26, 2021 at 11:41:56AM +0900, Stephen J. Turnbull wrote:
That's what I would mean by basic sys-admin skills. And *surprise!* my students don't have them, and don't need them ... until they start using Python.
Is it *only* Python though? Wouldn't that be necessary if they were learning Perl, Java, C, Ruby etc?
I'm sure it is. But as I say *my* students are pretty naive, so they only use languages supported by me (Python and R, but almost exclusively Python) in Jupyter and installed via Conda. Not a lot of folks around here can speak all of those, Japanese, and English, and read minds in Chinese.[1] Otherwise they use site-licensed commercial software (mostly SPSS, Mathematica, and office software). Footnotes: [1] This is the easiest part of all. It's not hard to guess what the students are whispering in Chinese when you've just prohibited use of Chinese in the seminar room. :-)
![](https://secure.gravatar.com/avatar/2dde9ab55441548a981af3e43639fcbd.jpg?s=120&d=mm&r=g)
On Tue, Feb 23, 2021 at 7:48 PM Random832 <random832@fastmail.com> wrote:
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
[snip]
How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?
I can't speak for distributors or maintainers [1], but I can speak for myself as a user. I run Debian testing (currently bullseye as that is preparing for release) as my daily OS on my personal laptop, used for personal matters and school assignments (I'm a university computer science student in my senior year). I don't use the system Python for anything of my own, whether it's a school assignment or a personal project, precisely because I don't want to risk screwing something up. Rather, I maintain a clone/fork of the official CPython GitHub repo, and periodically build from source and `make altinstall` into `~/.local/`. The `python3` command continues to refer to the system Python, while `python3.8`, etc. refer to the ones installed in my home folder. To the latter I make symlinks for `py38`, `py39`, etc., and just `py` (and `pip`) for the one I use most often (usually the latest stable release). I typically have multiple versions installed at once since different scripts/projects run on different versions at different times. Given this setup, I can just do a simple `pip install spam` command and I don't need either `sudo` or `--user`, nor do I need virtual envs. While the average person would probably not clone the GitHub repo and build that way, it's not terribly unreasonable for an inclined person to do the same with a tarball downloaded from python.org, and so I doubt I'm the only one with this type of setup. Just some food for thought. [1] Technically I am a library maintainer since I have a couple projects on PyPI, but those are mostly unused and more or less abandoned at this point, and neither ever reached the point where I could consider graduating them from beta status. Most of what I work with these days is private personal code or school assignments.
![](https://secure.gravatar.com/avatar/1fee087d7a1ca17c8ad348271819a8d5.jpg?s=120&d=mm&r=g)
On Tue, 23 Feb 2021 20:29:52 -0500 Jonathan Goble <jcgoble3@gmail.com> wrote:
I can't speak for distributors or maintainers [1], but I can speak for myself as a user. I run Debian testing (currently bullseye as that is preparing for release) as my daily OS on my personal laptop, used for personal matters and school assignments (I'm a university computer science student in my senior year).
I don't use the system Python for anything of my own, whether it's a school assignment or a personal project, precisely because I don't want to risk screwing something up. Rather, I maintain a clone/fork of the official CPython GitHub repo, and periodically build from source and `make altinstall` into `~/.local/`.
For the record, instead of building Python by hand, you could use a distribution such as Anaconda or the community-maintained conda-forge. Regards Antoine.
![](https://secure.gravatar.com/avatar/7cdb1a3f28807091044bdf203ecaca8b.jpg?s=120&d=mm&r=g)
On Wed, 24 Feb 2021 at 10:18, Antoine Pitrou <antoine@python.org> wrote:
On Tue, 23 Feb 2021 20:29:52 -0500 Jonathan Goble <jcgoble3@gmail.com> wrote:
I can't speak for distributors or maintainers [1], but I can speak for myself as a user. I run Debian testing (currently bullseye as that is preparing for release) as my daily OS on my personal laptop, used for personal matters and school assignments (I'm a university computer science student in my senior year).
I don't use the system Python for anything of my own, whether it's a school assignment or a personal project, precisely because I don't want to risk screwing something up. Rather, I maintain a clone/fork of the official CPython GitHub repo, and periodically build from source and `make altinstall` into `~/.local/`.
For the record, instead of building Python by hand, you could use a distribution such as Anaconda or the community-maintained conda-forge.
Regards
Antoine.
I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been quite happy with that, and it basically does what Jonathan does manually: clone the github repo and build python from scratch.
![](https://secure.gravatar.com/avatar/e91a97baa90dd7f5a267c79a2cbb04b2.jpg?s=120&d=mm&r=g)
On Wed, Feb 24, 2021 at 11:43 AM Henk-Jaap Wagenaar < wagenaarhenkjaap@gmail.com> wrote:
I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been quite happy with that, and it basically does what Jonathan does manually: clone the github repo and build python from scratch.
Also using pyenv myself, on platforms where either, or both, new and old version of Python are not available (so on Ubuntu I'm sticking with Anthony Sottile's deadsnakes.) But I'm not using it to manage virtual envs. For my own development projects, I'm using VirtualFish since I've switched to Fish as my main shell, but I was happy with VirtualenvWrapper when I was still using bash or zsh. I can't live without VirtualFish's autoactivation plugin (which activate a given virtualenv each time you enter your project). There was something similar for Zsh though I don't remember which. Sometimes I forget that I'm not in the right folder, or that I didn't create the virtualenv, or that I removed it in the case of an old project, and I end up polluting my default Python install after a 'pip install ...'. No big deal but always a bit unpleasant. And as I wrote before, I'm now using pipx to install the apps that I use. This works not so bad (except for some quirks now and then) but it took quite some time to select the right (for me) tools, to set up the proper config files on my computer, and it can be annoying when working on a server which has not been set up the same way. It also takes some time to explain to collaborators, especially interns or juniors. So obviously there is room for improvement, especially in terms of simplicity for beginners. S. _______________________________________________
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OZRHP3Q6... Code of Conduct: http://python.org/psf/codeofconduct/
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
![](https://secure.gravatar.com/avatar/fd079755731ad59ebedad02b690340e8.jpg?s=120&d=mm&r=g)
On Tue, Feb 23, 2021 at 4:51 PM Random832 <random832@fastmail.com> wrote:
Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
The alternative to virtualenv is the solution most software takes for, eg, Java: Package a complete binary distribution internally to eliminate any dependency on the system package -- and allow the software to pollute their own package in any way they want without affecting the rest of the system, while potentially becoming very out of date with security patches. Virtualenv allows software to lockstep and pollute their own runtime of Python almost all they want while still sharing the actual base install that gets regular security and bugfix updates. (Of course, even the hint of possibility that anything about the runtime could change and cause downtime is too much for many enterprise Java systems, no matter how severe the security update, so even ported to Python they wouldn't settle for that, they'd still package the entire interpreter internally.) Since telling application developers "no, don't do that, just get along" doesn't work, no matter how loudly you say it, virtualenv is a happy middle ground. A few core OS maintainers are much more likely to listen, so the yelling about splitting up or overly customizing the stdlib achieves results. -Em
![](https://secure.gravatar.com/avatar/63ca18e130d527d0741f1da54bb129a7.jpg?s=120&d=mm&r=g)
On 2/23/21, Random832 <random832@fastmail.com> wrote:
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.
How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?
First, pip+venv is not sufficient for secure software deployment: something must set appropriate permissions so that the application cannot overwrite itself and other core libraries (in order to eliminate W^X violations (which e.g. Android is solving by requiring all installed binaries to come from an APK otherwise they won't and can't be labeled with the SELinux extended file atrributes necessary for a binary to execute; but we don't have binaries, we have an interpreter and arbitrary hopefully-signed somewhere source code, at least)). Believe it or not, this is wrong: ```bash # python -m venv httpbin || virtualenv httpbin # source httpbin/bin/activate mkvirtualenv httpbin pip install httpbin gunicorn gunicorn -b 0.0.0.0:8001 httpbin:app # python -m webbrowser http://127.0.0.1:8001 ``` It's wrong - it's insecure - because the user executing the Python interpreter (through gunicorn, in this case) can overwrite the app. W^X: has both write and execute permissions. What would be better? This would be better because pip isn't running setup.py as root (with non-wheels) and httpbin_exec can't modify the app interpreter or the code it loads at runtime: ```bash useradd httpbin # also creates a group also named 'httpbin' sudo -u httpbin sh -c ' \ python -m venv httpbin; \ umask 0022; \ ./httpbin/bin/python -m pip install httpbin gunicorn' useradd httpbin_exec -G httpbin sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app' ``` This would be better if it worked, though there are a few caveats: ```bash sudo apt-get install python-gunicorn python-httpbin sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app ``` 1. Development is impossible: - You can't edit the code in /usr/lib/python3.n/site-package/ without root permissions. - You should not be running an editor as root. - You can edit distro-package files individually with e.g. sudoedit (and then the GPG-signed package file checksums will fail when you run `debsums` or `rpm -Va` because you've edited the file and that's changed the hash). - Non-root users cannot install python packages without having someone repack (and sign it) for them. - What do I need to do in order to patch the distro's signed repack of the Python package released to PyPI? - I like how Fedora pkgs and conda-forge have per-package git repos now. - Conda-forge has a bot that watches PyPI for new releases and tries sending an automated PR. - If I send a PR to the main branch of the source repo and it gets merged, how long will it be before there's a distro repack built and uploaded to the distro package index? 2. It should be installed in a chroot/jail/zone/container/context/vm so that it cannot read other data on the machine. The httpbin app does not need read access to /etc/shadow, for example. Distro package installs are not - either - sandboxed. To pick on httpbin a bit more, the httpbin docs specify that httpbin should be run as a docker container: ```bash docker run -p 80:8001 kennethreitz/httpbin ``` Is that good enough? We don't know, we haven't reviewed: - the Dockerfile - it says `FROM ubuntu:18.04`, which is fortunately an LTS release. But if it hasn't been updated this month, it probably has the sudo bug that enabled escalation to root (which - even in a container - is bad because it could obnoxiously just overwrite libc, for example, and unless the container is rebuilt or something runs `debsums`, nothing will detect that data integrity error) - the requirements.txt / setup.py:install_requires / Pipfile[.lock] dependencies - does it depend upon outdated pinned exact versions? - Is there an SBOM (Software Bill of Materials) that we can review against known vulnerability databases? How do I know that: - The packages I have installed are not outdated and unpatched against known vulnerabilities - The files on disk are exactly what should be in the package - The app_exec user can't overwrite the binary interpreter or the source files it loads at runtime - There won't be unreviewed code running as root (including at install time) - All Python package dependencies are available as wheels (that basically only need to be unzipped) - The ensemble of dependencies which I've miraculously assembled is available on the target platform(s) - The integration tests for my app pass with each combination of dependencies which satisfy the specified dependency constraints - I can edit things and quickly re-test - Each dependency is signed by a key that's valid for that dependency So, if pip is insufficient for secure software deployment, what are pro teams using to build signed, deployable artifacts with fresh, upgaded dependencies either bundled in or loosely-referenced? - Bazel (from Google's internal Blaze) builds from BUILD files. - https://github.com/dropbox/dbx_build_tools - Pantsbuild, Buck - zipapps - FPM can apparently package up an entire virtualenv; though IDK how good it is at permissions? https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/virtualenv.r... As an open source maintainer, there are very many potential environments to release builds for. Manylinux docker images (and auditwheel, delocate, and *cibuildwheel*) are a response to extreme and somewhat-avoidable complexity. https://github.com/joerick/cibuildwheel Distro packagers can and do build upon e.g. pip; which is great for development but not sufficient for production deployment due to lack of support for file permissions, extended file attributes, checksums, cryptographic signatures, and due to running setup.py as the install user for non-wheel packages. There are many deployment stories now: pull/push, configuration management systems, venvs within containers within VMs. For your favorite distro, how do I get from cibuildwheel to a signed release artifact in your package index; and which keys can sign for what?
![](https://secure.gravatar.com/avatar/63ca18e130d527d0741f1da54bb129a7.jpg?s=120&d=mm&r=g)
FWIW, Distro repacks are advantageous in comparison to "statically-bundled" releases that for example bundle in an outdated version of OpenSSL, because when you `apt-get upgrade -y` that should upgrade the OpenSSL that all the other distro packages depend upon. Here's something that doesn't get called as frequently as `apt-get upgrade -y`: ```bash pip install -U certifi ``` https://github.com/certifi/python-certifi (Mozilla's CA bundle extracted into a Python package) ```bash apt-get install -y ca-certificates dnf install -y ca-certificates ``` On 2/24/21, Wes Turner <wes.turner@gmail.com> wrote:
On 2/23/21, Random832 <random832@fastmail.com> wrote:
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.
How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?
First, pip+venv is not sufficient for secure software deployment: something must set appropriate permissions so that the application cannot overwrite itself and other core libraries (in order to eliminate W^X violations (which e.g. Android is solving by requiring all installed binaries to come from an APK otherwise they won't and can't be labeled with the SELinux extended file atrributes necessary for a binary to execute; but we don't have binaries, we have an interpreter and arbitrary hopefully-signed somewhere source code, at least)).
Believe it or not, this is wrong:
```bash # python -m venv httpbin || virtualenv httpbin # source httpbin/bin/activate mkvirtualenv httpbin
pip install httpbin gunicorn gunicorn -b 0.0.0.0:8001 httpbin:app
# python -m webbrowser http://127.0.0.1:8001 ```
It's wrong - it's insecure - because the user executing the Python interpreter (through gunicorn, in this case) can overwrite the app. W^X: has both write and execute permissions. What would be better?
This would be better because pip isn't running setup.py as root (with non-wheels) and httpbin_exec can't modify the app interpreter or the code it loads at runtime:
```bash useradd httpbin # also creates a group also named 'httpbin' sudo -u httpbin sh -c ' \ python -m venv httpbin; \ umask 0022; \ ./httpbin/bin/python -m pip install httpbin gunicorn'
useradd httpbin_exec -G httpbin sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app' ```
This would be better if it worked, though there are a few caveats:
```bash sudo apt-get install python-gunicorn python-httpbin sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app ```
1. Development is impossible: - You can't edit the code in /usr/lib/python3.n/site-package/ without root permissions. - You should not be running an editor as root. - You can edit distro-package files individually with e.g. sudoedit (and then the GPG-signed package file checksums will fail when you run `debsums` or `rpm -Va` because you've edited the file and that's changed the hash).
- Non-root users cannot install python packages without having someone repack (and sign it) for them.
- What do I need to do in order to patch the distro's signed repack of the Python package released to PyPI? - I like how Fedora pkgs and conda-forge have per-package git repos now. - Conda-forge has a bot that watches PyPI for new releases and tries sending an automated PR. - If I send a PR to the main branch of the source repo and it gets merged, how long will it be before there's a distro repack built and uploaded to the distro package index?
2. It should be installed in a chroot/jail/zone/container/context/vm so that it cannot read other data on the machine. The httpbin app does not need read access to /etc/shadow, for example. Distro package installs are not - either - sandboxed.
To pick on httpbin a bit more, the httpbin docs specify that httpbin should be run as a docker container:
```bash docker run -p 80:8001 kennethreitz/httpbin ```
Is that good enough? We don't know, we haven't reviewed:
- the Dockerfile - it says `FROM ubuntu:18.04`, which is fortunately an LTS release. But if it hasn't been updated this month, it probably has the sudo bug that enabled escalation to root (which - even in a container - is bad because it could obnoxiously just overwrite libc, for example, and unless the container is rebuilt or something runs `debsums`, nothing will detect that data integrity error) - the requirements.txt / setup.py:install_requires / Pipfile[.lock] dependencies - does it depend upon outdated pinned exact versions? - Is there an SBOM (Software Bill of Materials) that we can review against known vulnerability databases?
How do I know that:
- The packages I have installed are not outdated and unpatched against known vulnerabilities - The files on disk are exactly what should be in the package - The app_exec user can't overwrite the binary interpreter or the source files it loads at runtime - There won't be unreviewed code running as root (including at install time) - All Python package dependencies are available as wheels (that basically only need to be unzipped) - The ensemble of dependencies which I've miraculously assembled is available on the target platform(s) - The integration tests for my app pass with each combination of dependencies which satisfy the specified dependency constraints - I can edit things and quickly re-test - Each dependency is signed by a key that's valid for that dependency
So, if pip is insufficient for secure software deployment, what are pro teams using to build signed, deployable artifacts with fresh, upgaded dependencies either bundled in or loosely-referenced?
- Bazel (from Google's internal Blaze) builds from BUILD files. - https://github.com/dropbox/dbx_build_tools - Pantsbuild, Buck - zipapps - FPM can apparently package up an entire virtualenv; though IDK how good it is at permissions?
https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/virtualenv.r...
As an open source maintainer, there are very many potential environments to release builds for. Manylinux docker images (and auditwheel, delocate, and *cibuildwheel*) are a response to extreme and somewhat-avoidable complexity. https://github.com/joerick/cibuildwheel
Distro packagers can and do build upon e.g. pip; which is great for development but not sufficient for production deployment due to lack of support for file permissions, extended file attributes, checksums, cryptographic signatures, and due to running setup.py as the install user for non-wheel packages.
There are many deployment stories now: pull/push, configuration management systems, venvs within containers within VMs. For your favorite distro, how do I get from cibuildwheel to a signed release artifact in your package index; and which keys can sign for what?
-- Wes Turner https://westurner.org https://wrdrd.com/docs/consulting/knowledge-engineering
![](https://secure.gravatar.com/avatar/ff5429ef741b434eb2011692832f810e.jpg?s=120&d=mm&r=g)
On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.
The problem is a little deeper and the Debian solution doesn't really solve all of it. Yes, pip installing into the same directory as the package manager is a problem. It's a problem to the point that I'm patching pip in Gentoo to explicitly block that. We've gotten far too many bug reports about people's systems suddenly being horribly broken after they used pip. While using two directories can prevent pip from directly overwriting system packages, you still can't expect two independent package managers to work simultaneously unless they can communicate with each other to prevent conflicts. If pip installs a different version of the same package as the package manager, which one is supposed to be used? Whichever choice you make, you'll bound to eventually break dependency graph of some package. -- Best regards, Michał Górny
![](https://secure.gravatar.com/avatar/33bd15feb2558d0050e863875e0f5f60.jpg?s=120&d=mm&r=g)
On 24/02/2021 08.03, Michał Górny wrote:
On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.
The problem is a little deeper and the Debian solution doesn't really solve all of it.
Yes, pip installing into the same directory as the package manager is a problem. It's a problem to the point that I'm patching pip in Gentoo to explicitly block that. We've gotten far too many bug reports about people's systems suddenly being horribly broken after they used pip.
While using two directories can prevent pip from directly overwriting system packages, you still can't expect two independent package managers to work simultaneously unless they can communicate with each other to prevent conflicts. If pip installs a different version of the same package as the package manager, which one is supposed to be used? Whichever choice you make, you'll bound to eventually break dependency graph of some package.
Thanks Michał, you have explained the situation well. Most Linux distros have two separate directories for distro packages and "sudo pip install" packages. This is a great thing! Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system. The package manager of several Linux distributions is written in Python. If your system Python is broken, your package manager is likely busted as well. The split directories allow ysers to fix their system with a single command like "sudo rm -rf /usr/local/lib/python*" (paths may vary). The issue is: Split distro/pip directories are not baked into Python or pip. Distros have their own set of downstream patches that modify the behavior of Python. Some of the patches have caused issues in the past. I would like to see this feature implemented in Python's core, so distro packages have to do less patching and upstream pip can test against it. Christian PS: I wrote PEP 370 for "install --user" 13 years ago before we had virtual envs.
![](https://secure.gravatar.com/avatar/3dd475b8aaa5292d74cb0c3f76c3f196.jpg?s=120&d=mm&r=g)
On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.
I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path. Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installation.
![](https://secure.gravatar.com/avatar/33bd15feb2558d0050e863875e0f5f60.jpg?s=120&d=mm&r=g)
On 24/02/2021 15.16, Random832 wrote:
On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.
I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.
Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installation.
Yes, you are describing one major aspect of my idea for a system Python interpreter. I'm happy to read that other users are coming to similar conclusions. Instead of an option I'd create a new executable to lock down additional things (e.g. isolated mode, code verification hook). A separate executable would also allow distros to provide a stripped down interpreter that does not cause bad user experience.
![](https://secure.gravatar.com/avatar/be200d614c47b5a4dbb6be867080e835.jpg?s=120&d=mm&r=g)
On 2/24/2021 4:26 PM, Christian Heimes wrote:
On 24/02/2021 15.16, Random832 wrote:
On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.
I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.
Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installation.
Yes, you are describing one major aspect of my idea for a system Python interpreter. I'm happy to read that other users are coming to similar conclusions. Instead of an option I'd create a new executable to lock down additional things (e.g. isolated mode, code verification hook). A separate executable would also allow distros to provide a stripped down interpreter that does not cause bad user experience.
I mean, this is _precisely_ what PEP 370 defines (including the "-s" option and PYTHONNOUSERSITE env variable to provide that one-way isolation). Is the problem that pip doesn't use it by default? Or that the distros went and made patches for the runtime rather than patching pip? (For Windows installs from the Store, where even admin rights can't do an all-users package install, I added a new config file location for pip to change this default, but a patch would also have worked.) Maybe we need an easier way to patch the location of user site packages? I also had to do this for the Store install on Windows, and it's a little bit of a hack... but maybe having an official recommendation would help encourage distributors to use the mechanism? Cheers, Steve
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Wed, 24 Feb 2021 at 18:25, Steve Dower <steve.dower@python.org> wrote:
I mean, this is _precisely_ what PEP 370 defines (including the "-s" option and PYTHONNOUSERSITE env variable to provide that one-way isolation).
Is the problem that pip doesn't use it by default?
Note that these days, pip *does* use usersite by default if site-packages isn't writable. Of course, if users run pip under sudo, that invalidates that mechanism :-( Paul
![](https://secure.gravatar.com/avatar/33bd15feb2558d0050e863875e0f5f60.jpg?s=120&d=mm&r=g)
On 24/02/2021 19.17, Steve Dower wrote:
On 2/24/2021 4:26 PM, Christian Heimes wrote:
On 24/02/2021 15.16, Random832 wrote:
On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.
I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.
Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installati on.
Yes, you are describing one major aspect of my idea for a system Python interpreter. I'm happy to read that other users are coming to similar conclusions. Instead of an option I'd create a new executable to lock down additional things (e.g. isolated mode, code verification hook). A separate executable would also allow distros to provide a stripped down interpreter that does not cause bad user experience.
I mean, this is _precisely_ what PEP 370 defines (including the "-s" option and PYTHONNOUSERSITE env variable to provide that one-way isolation).
Is the problem that pip doesn't use it by default? Or that the distros went and made patches for the runtime rather than patching pip? (For Windows installs from the Store, where even admin rights can't do an all-users package install, I added a new config file location for pip to change this default, but a patch would also have worked.)
Maybe we need an easier way to patch the location of user site packages? I also had to do this for the Store install on Windows, and it's a little bit of a hack... but maybe having an official recommendation would help encourage distributors to use the mechanism?
(sorry for terse reply, I'm heading out.) Linux distros want an additional layer so the can differentiate between Python packages installed by their package manager and Python packages installed with "sudo pip install". Python has two default site-package directories (from highest to lowest precedence): 1. ~/.local/lib/python3.9/site-packages 2. /usr/lib/python3.9/site-packages (1) is defined by site.getusersitepackages(), (2) site.getsitepackages() Linux distro have additional site-package directories. My Fedora system X86_64 system with multiarch has three additional candidates. 1. ~/.local/lib/python3.9/site-packages 2. /usr/local/lib64/python3.9/site-packages 3. /usr/local/lib/python3.9/site-packages 4. /usr/lib64/python3.9/site-packages 5. /usr/lib/python3.9/site-packages The "lib" directories are for pure Python packages whiel the "lib64" directories contain X86_64 native code extensions (aka shared libraries). The "/usr/lib*" directories are used for distro packages. The downstream patch [1] ensures that "sudo pip install" installs packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used. Debian has a similar mechanism to provide "/usr/lib/python3/dist-packages" [2]. A hypothetical /usr/bin/system-python3 interpreter would only use (4) and (5) as site-package directories. A user-facing /usr/bin/python3 would use (1) to (5). Christian [1] https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-use... [2] https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html
![](https://secure.gravatar.com/avatar/33bd15feb2558d0050e863875e0f5f60.jpg?s=120&d=mm&r=g)
On 24/02/2021 20.03, Christian Heimes wrote:
On 24/02/2021 19.17, Steve Dower wrote:
On 2/24/2021 4:26 PM, Christian Heimes wrote:
On 24/02/2021 15.16, Random832 wrote:
On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.
I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.
Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installati on.
Yes, you are describing one major aspect of my idea for a system Python interpreter. I'm happy to read that other users are coming to similar conclusions. Instead of an option I'd create a new executable to lock down additional things (e.g. isolated mode, code verification hook). A separate executable would also allow distros to provide a stripped down interpreter that does not cause bad user experience.
I mean, this is _precisely_ what PEP 370 defines (including the "-s" option and PYTHONNOUSERSITE env variable to provide that one-way isolation).
Is the problem that pip doesn't use it by default? Or that the distros went and made patches for the runtime rather than patching pip? (For Windows installs from the Store, where even admin rights can't do an all-users package install, I added a new config file location for pip to change this default, but a patch would also have worked.)
Maybe we need an easier way to patch the location of user site packages? I also had to do this for the Store install on Windows, and it's a little bit of a hack... but maybe having an official recommendation would help encourage distributors to use the mechanism?
(sorry for terse reply, I'm heading out.)
Linux distros want an additional layer so the can differentiate between Python packages installed by their package manager and Python packages installed with "sudo pip install".
Python has two default site-package directories (from highest to lowest precedence):
1. ~/.local/lib/python3.9/site-packages 2. /usr/lib/python3.9/site-packages
(1) is defined by site.getusersitepackages(), (2) site.getsitepackages()
Linux distro have additional site-package directories. My Fedora system X86_64 system with multiarch has three additional candidates.
1. ~/.local/lib/python3.9/site-packages 2. /usr/local/lib64/python3.9/site-packages 3. /usr/local/lib/python3.9/site-packages 4. /usr/lib64/python3.9/site-packages 5. /usr/lib/python3.9/site-packages
The "lib" directories are for pure Python packages whiel the "lib64" directories contain X86_64 native code extensions (aka shared libraries). The "/usr/lib*" directories are used for distro packages. The downstream patch [1] ensures that "sudo pip install" installs packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.
Debian has a similar mechanism to provide "/usr/lib/python3/dist-packages" [2].
A hypothetical /usr/bin/system-python3 interpreter would only use (4) and (5) as site-package directories. A user-facing /usr/bin/python3 would use (1) to (5).
Christian
[1] https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-use... [2] https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html
PS: Fedora patch [1] uses -s / PYTHONNOUSERSITE to exclude "/usr/local/lib*" site-package directories. $ python3 -c 'import site; print(site.getsitepackages())' ['/usr/local/lib64/python3.9/site-packages', '/usr/local/lib/python3.9/site-packages', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages'] $ python3 -s -c 'import site; print(site.getsitepackages())' ['/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']
![](https://secure.gravatar.com/avatar/63ca18e130d527d0741f1da54bb129a7.jpg?s=120&d=mm&r=g)
Would it be better to have an (os, os_release, arch) to [site_packages_dirs] map within CPython or are the patches too varied to be replaced by a centralized map and a runtime conditional? For development convenience, having writeable site-packages dirs before unwriteable site-packages is easy; But for security and production deployment, site-packages maybe shouldn't be writeable at all by default because that's doing it wrong? On Wed, Feb 24, 2021, 15:34 Christian Heimes <christian@python.org> wrote:
On 24/02/2021 20.03, Christian Heimes wrote:
On 24/02/2021 19.17, Steve Dower wrote:
On 2/24/2021 4:26 PM, Christian Heimes wrote:
On 24/02/2021 15.16, Random832 wrote:
On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.
I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.
Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installati on.
Yes, you are describing one major aspect of my idea for a system Python interpreter. I'm happy to read that other users are coming to similar conclusions. Instead of an option I'd create a new executable to lock down additional things (e.g. isolated mode, code verification hook). A separate executable would also allow distros to provide a stripped down interpreter that does not cause bad user experience.
I mean, this is _precisely_ what PEP 370 defines (including the "-s" option and PYTHONNOUSERSITE env variable to provide that one-way isolation).
Is the problem that pip doesn't use it by default? Or that the distros went and made patches for the runtime rather than patching pip? (For Windows installs from the Store, where even admin rights can't do an all-users package install, I added a new config file location for pip to change this default, but a patch would also have worked.)
Maybe we need an easier way to patch the location of user site packages? I also had to do this for the Store install on Windows, and it's a little bit of a hack... but maybe having an official recommendation would help encourage distributors to use the mechanism?
(sorry for terse reply, I'm heading out.)
Linux distros want an additional layer so the can differentiate between Python packages installed by their package manager and Python packages installed with "sudo pip install".
Python has two default site-package directories (from highest to lowest precedence):
1. ~/.local/lib/python3.9/site-packages 2. /usr/lib/python3.9/site-packages
(1) is defined by site.getusersitepackages(), (2) site.getsitepackages()
Linux distro have additional site-package directories. My Fedora system X86_64 system with multiarch has three additional candidates.
1. ~/.local/lib/python3.9/site-packages 2. /usr/local/lib64/python3.9/site-packages 3. /usr/local/lib/python3.9/site-packages 4. /usr/lib64/python3.9/site-packages 5. /usr/lib/python3.9/site-packages
The "lib" directories are for pure Python packages whiel the "lib64" directories contain X86_64 native code extensions (aka shared libraries). The "/usr/lib*" directories are used for distro packages. The downstream patch [1] ensures that "sudo pip install" installs packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.
Debian has a similar mechanism to provide "/usr/lib/python3/dist-packages" [2].
A hypothetical /usr/bin/system-python3 interpreter would only use (4) and (5) as site-package directories. A user-facing /usr/bin/python3 would use (1) to (5).
Christian
[1]
https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-use...
[2]
https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html
PS: Fedora patch [1] uses -s / PYTHONNOUSERSITE to exclude "/usr/local/lib*" site-package directories.
$ python3 -c 'import site; print(site.getsitepackages())' ['/usr/local/lib64/python3.9/site-packages', '/usr/local/lib/python3.9/site-packages', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']
$ python3 -s -c 'import site; print(site.getsitepackages())' ['/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EKYM66AZ... Code of Conduct: http://python.org/psf/codeofconduct/
![](https://secure.gravatar.com/avatar/f3ba3ecffd20251d73749afbfa636786.jpg?s=120&d=mm&r=g)
On Wed, 24 Feb 2021 at 10:49, Random832 <random832@fastmail.com> wrote:
I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
The reason venv is promoted as heavily as it is is because it's the only advice that can be given that is consistently correct regardless of the operating system the user is running locally, whereas safely using a system-wide Python installation varies a lot depending on whether you're on Windows, Mac OS X, or Linux (let alone some other platform outside the big 3 desktop clients). conda is also popular for the same reason: while the instructions for installing conda in the first place are OS-dependent, once it is up and running you can use consistent platform independent conda commands rather than having to caveat all your documentation with platform-specific instructions. Apple moved all of their dynamic language interpreter implementations to inaccessible-by-default locations so Mac OS X users would stop using them to run their own code. Alongside that, we *have* worked with the Linux distro vendors to help make "sudo pip install" safe (e.g [1]), but that only helps if a user is running a new enough version of a distro that has participated in that work. However, while the option of running "platform native" environments will never go away, and work will continue to make it less error prone, the level of knowledge of your specific OS's idiosyncrasies that it requires is almost certainly going to remain too high for it to ever again become the default recommendation that it used to be. Cheers, Nick. [1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note: this change mitigated some aspects of the problem in a way similar to what Debian does, but still doesn't solve it completely, as custom Python builds may still make arbitrary changes) P.S. "But what about user site-packages?" you ask. Until relatively recently, Debian didn't put the user's local bin directory on the system path by default, so commands provided by user level package installs didn't work without the user adjusting their PATH. The CPython Windows installer also doesn't adjust PATH by default (for good reasons). And unlike a venv, "python -m" doesn't let you ensure that the code executed is the version installed in user site-packages - it could be coming from a directory earlier in sys.path. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
![](https://secure.gravatar.com/avatar/d91ce240d2445584e295b5406d12df70.jpg?s=120&d=mm&r=g)
And unlike a venv, "python -m" doesn't let you ensure that the code executed is the version installed in user site-packages
I have had enough problems with this that when I do modify/replace something, I put in a marker that I can check for explicitly. Expecting this marker to run automatically at import (and the results not to be swallowed) turned out to be insufficient.
it could be coming from a directory earlier in sys.path.
The obvious solution is to put the current directory in front of sys.path. Alas, security folks didn't like that idea.
![](https://secure.gravatar.com/avatar/63ca18e130d527d0741f1da54bb129a7.jpg?s=120&d=mm&r=g)
Is there a tool that (1) detects import name collisions; and (2) attempts to read package metadata and package file checksums (maybe from the ZIP 'manifest')? In order to: - troubleshoot module shadowing issues - $PATH - sys.path - `python -m site` - incomplete and overlapping uninstallations: pip install a pip install a_modified # pip uninstall a? pip install pdbpp pip uninstall a_modified ls -altr "${site-packages[*]}" strace -e trace=file python -c 'import pdb' **** When shouldn't site customizations be added to the site module? https://docs.python.org/3/library/site.html When should customizations be built into the build instead of a runtime conditional? On Sat, Feb 27, 2021, 23:12 Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, 24 Feb 2021 at 10:49, Random832 <random832@fastmail.com> wrote:
I was reading a discussion thread <
https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
Christian Heimes wrote:
Core dev and PyPA has spent a lot of effort in promoting venv because
we don't want users to break their operating system with sudo pip install.
I don't think sudo pip install should break the operating system. And I
think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
The reason venv is promoted as heavily as it is is because it's the only advice that can be given that is consistently correct regardless of the operating system the user is running locally, whereas safely using a system-wide Python installation varies a lot depending on whether you're on Windows, Mac OS X, or Linux (let alone some other platform outside the big 3 desktop clients).
conda is also popular for the same reason: while the instructions for installing conda in the first place are OS-dependent, once it is up and running you can use consistent platform independent conda commands rather than having to caveat all your documentation with platform-specific instructions.
Apple moved all of their dynamic language interpreter implementations to inaccessible-by-default locations so Mac OS X users would stop using them to run their own code.
Alongside that, we *have* worked with the Linux distro vendors to help make "sudo pip install" safe (e.g [1]), but that only helps if a user is running a new enough version of a distro that has participated in that work.
However, while the option of running "platform native" environments will never go away, and work will continue to make it less error prone, the level of knowledge of your specific OS's idiosyncrasies that it requires is almost certainly going to remain too high for it to ever again become the default recommendation that it used to be.
Cheers, Nick.
[1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note: this change mitigated some aspects of the problem in a way similar to what Debian does, but still doesn't solve it completely, as custom Python builds may still make arbitrary changes)
P.S. "But what about user site-packages?" you ask. Until relatively recently, Debian didn't put the user's local bin directory on the system path by default, so commands provided by user level package installs didn't work without the user adjusting their PATH. The CPython Windows installer also doesn't adjust PATH by default (for good reasons). And unlike a venv, "python -m" doesn't let you ensure that the code executed is the version installed in user site-packages - it could be coming from a directory earlier in sys.path.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RDLEH6DU... Code of Conduct: http://python.org/psf/codeofconduct/
participants (21)
-
Antoine Pitrou
-
Christian Heimes
-
Emily Bowman
-
Eryk Sun
-
Fred Drake
-
Henk-Jaap Wagenaar
-
Jim J. Jewett
-
Jonathan Goble
-
Michał Górny
-
Mike Miller
-
Nick Coghlan
-
Oscar Benjamin
-
Paul Bryan
-
Paul Moore
-
Peter Wang
-
Random832
-
Stephen J. Turnbull
-
Steve Dower
-
Steven D'Aprano
-
Stéfane Fermigier
-
Wes Turner