On 24/02/2021 11.52, Stéfane Fermigier wrote:

I love pipx and I'm glad it exists at this point because it make

The main issue is that each virtualenv takes space, lots of space.

I have currently 57 apps installed via pipx on my laptop, and the 57 environments take almost 1 GB already.

~  cd .local/pipx/venvs/  ~/.l/p/venvs  ls abilian-tools/  concentration/  gitlabber/      pygount/        sphinx/ ansible/        cookiecutter/   httpie/         pyinfra/        tentakel/ assertize/      cruft/          isort/          pylint/         tlv/ autoflake/      cython/         jupyterlab/     pyre-check/     towncrier/ black/          dephell/        lektor/         pytype/         tox/ borgbackup/     docformatter/   md2pdf/         pyupgrade/      twine/ borgmatic/      flake8/         medikit/        radon/          virtualenv/ bpytop/         flit/           mypy/           re-ver/         virtualfish/ check-manifest/ flynt/          nox/            sailboat/       vulture/ clone-github/   gh-clone/       pdoc3/          salvo/ cloneall/       ghtop/          pdocs/          shed/ com2ann/        gitchangelog/   pybetter/       sixer/  ~/.l/p/venvs  du -sh . 990M.  ~/.l/p/venvs  ls | wc       57      57     475

There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations.

There are tools like https://rdfind.pauldreik.se/rdfind.1.html that create hard links to deduplicate files. Some files systems have deduplicated baked in, too.

Christian

On Wed, Feb 24, 2021 at 12:42 PM Paul Moore p.f.moore@gmail.com wrote:

On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier sf@fermigier.com wrote:

...

There is probably a clever way to reuse common packages (probably via

clever symlinking) and reduce the footprint of these installations.

Ultimately the problem is that a general tool can't deal with conflicts (except by raising an error). If application A depends on lib==1.0 and application B depends on lib==2.0, you simply can't have a (consistent) environment that supports both A and B. But that's the rare case - 99% of the time, there are no conflicts. One env per app is a safe, but heavy handed, approach. Managing environments manually isn't exactly *hard*, but it's annoying manual work that pipx does an excellent job of automating, so it's a disk space vs admin time trade-off.

There are three ways to approach the question:

1) Fully isolated envs. The safest option but uses the most space.

2) Try to minimise the number of dependencies installed by interpreting the requirements specification in the looser way possible. This is both algorithmically hard (see https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and risky, as you've noted.

3) But the best way IMHO is to compute dependencies for each virtualenv independently from the others, but still share the packages, using some indirection mechanisms (hard links, symlinks or some Python-specific constructs) when the versions match exactly.

The 3rd solution is probably the best of the 3, but the sharing mechanism still needs to be specified (and, if needed, implemented) properly.

I've tried Christian's suggestions of using rdfind on my pipx installation, and it claims to reduce the footprint by 30% (nice, but less than I expected. This would however scale better with the number of installed packages).

I'm not sure this would be practical in reality, OTOH, because I think there is a serious risk of breakage each time I would upgrade one of the packages (via 'pipx upgrade-all' for instance).

So IMHO the best way to implement solution 3 would be by using some variant of the approach popularized by Nix (repository of immutable packages + links to each virtualenv).

S.

On 2021-02-24 02:52, Stéfane Fermigier wrote:

I have currently 57 apps installed via pipx on my laptop, and the 57 environments take almost 1 GB already.

I never understood the fear around version conflicts. Perhaps it has to do with the decline of sys-admin skills over the years? So, the strategy above feels like swatting a fly with a sledgehammer to me. Same as with a venv for every app. :-)

Rather than installing every package this way, why not wait until a conflict actually occurs?

Personally, I rarely have such conflicts, maybe every few years or so. When it happens, I fix them by uninstalling the offender and putting the more difficult one into the venv or pipx. Right now I only have one, a giant app from work that uses pipenv, and it's fine.

Now what about sudo and all that? Well, I used it in the old days because that's what the instructions said. But, to be honest, it never made any sense. I haven't shared a computer in decades, and when we did we used NFS for common tools, so it never saved any disk space.

Pip (and easy_install?) dragged their feet for years to properly support user installs (should have been default) but once they did I didn't look back. I dump almost all packages to user, which gets cleaned up every other year when the distro moves to the next version. The strategy has been working well for a long time.

So, --user works at the low end, and containers for the high end. Honestly, I don't have much of a use case any longer for venvs.

-Mike

The challenge with version conflicts is often less that you need to go update the constraints (which has little to do with sysadmin'ing, TBH) and more that you have insufficient *integration tests* and you're relying upon something else running the per-package tears.

On Thu, Feb 25, 2021, 00:10 Peter Wang pwang@anaconda.com wrote:

On Wed, Feb 24, 2021 at 8:50 PM Mike Miller python-dev@mgmiller.net wrote:

...

I never understood the fear around version conflicts.

With binary extension modules, version conflicts lead to (at best) runtime segfault and (at worst) subtle *data* bugs that return incorrect results. There are also deeper concerns around security and reproducibility.

...

Perhaps it has to do with the decline of sys-admin skills over the years?

Many millions of users of new Python users show up every year, using the language and its powerful ecosystem for data analytics and scientific computing, and they have no hope of having sys-admin skills.

-Peter

---

Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TYZ6QT4B... Code of Conduct: http://python.org/psf/codeofconduct/

One aspect of conda packaging that's worth adopting is the 'test' block of the meta.yaml which contains commands necessary to *test* the built package.

You can add a pytest command to setup.py, but it's not as easy as: https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml... :

```yaml test: downstreams: - cython - setuptools requires: - ripgrep - cmake - ninja - {{ compiler('c') }} # Tried to use enable_language(C) to avoid needing this. It does not work. - {{ compiler('cxx') }} files: - tests/distutils/* - tests/cmake/* - tests/cython/* - tests/prefix-replacement/* commands: - echo on # [win] - set # [win] - python -V - python3 -V # [not win] - 2to3 -h - pydoc -h - python3-config --help # [not win] - set "PIP_NO_BUILD_ISOLATION=False" # [win] - set "PIP_NO_DEPENDENCIES=True" # [win] - set "PIP_IGNORE_INSTALLED=True" # [win] - set "PIP_NO_INDEX=True" # [win] - set "PIP_CACHE_DIR=%CONDA_PREFIX%/pip_cache" # [win] - set "TEMP=%CONDA_PREFIX%/tmp" # [win] - mkdir "%TEMP%" # [win] - python -Im ensurepip --upgrade --default-pip # [win] # tzdata/zoneinfo test that will need the tzdata package to pass - python -c "from zoneinfo import ZoneInfo; from datetime import datetime; dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo('America/Los_Angeles')); print(dt.tzname())" - python -m venv test-venv - python -c "import sysconfig; print(sysconfig.get_config_var('CC'))" # [not win] - _CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu python -c "import sysconfig; print(sysconfig.get_config_var('CC'))" # [linux64] # check for unreplaced @ symbols in sysconfig files - for f in ${CONDA_PREFIX}/lib/python*/_sysconfig*.py; do echo "Checking $f:"; if [[ `rg @ $f` ]]; then echo "FAILED ON $f"; cat $f; exit 1; fi; done # [linux64 or osx] - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.a # [unix] - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.nolto.a # [unix] # https://github.com/conda-forge/python-feedstock/issues/384 - if exist %PREFIX%\Scripts\pydoc exit 1 # [win] - if exist %PREFIX%\Scripts\idle exit 1 # [win] - if exist %PREFIX%\Scripts\2to3 exit 1 # [win] - if not exist %PREFIX%\Scripts\pydoc-script.py exit 1 # [win] - if not exist %PREFIX%\Scripts\idle-script.py exit 1 # [win] - if not exist %PREFIX%\Scripts\2to3-script.py exit 1 # [win] - if not exist %PREFIX%\Scripts\idle.exe exit 1 # [win] - if not exist %PREFIX%\Scripts\2to3.exe exit 1 # [win] - if not exist %PREFIX%\Scripts\pydoc.exe exit 1 # [win] - pushd tests - pushd distutils - python setup.py install -v -v - python -c "import foobar" - popd - pushd prefix-replacement # [unix] - bash build-and-test.sh # [unix] - popd # [unix] - pushd cmake - cmake -GNinja -DPY_VER={{ version }} # --trace --debug-output --debug-trycompile . - popd - popd - test ! -f default.profraw # [osx] ```

In which package you then store integration tests for your whole application and it's integrated dependencies is then up to you. Unfortunately, we often don't include tests/ in our packages, so it's literally not possible to run the tests in production even if you have pytest_requires installed.

Perhaps distros would do well to implement support for integration and per-package tests.

On Thu, Feb 25, 2021, 12:55 Fred Drake fred@fdrake.net wrote:

On Thu, Feb 25, 2021 at 5:35 AM Wes Turner wes.turner@gmail.com wrote:

...

The challenge with version conflicts is often less that you need to go update the constraints (which has little to do with sysadmin'ing, TBH) and more that you have insufficient *integration tests* and you're relying upon something else running the per-package tears.

Sometimes, auto-correct really does understand!

Your point is right on target, and really can't be understated.

-Fred

-- Fred L. Drake, Jr. <fred at fdrake.net> "There is nothing more uncommon than common sense." --Frank Lloyd Wright

Mike Miller writes:

"sys-admin" is a bit of an overstatement in my phrasing. The core is that you need to understand how a PATH works and be able to run pip and cd into folders and perhaps run rm/del, etc. Basic command-line skills?

That's what I would mean by basic sys-admin skills. And *surprise!* my students don't have them, and don't need them ... until they start using Python. They don't even understand installing packages in conda.

If I had my druthers, we wouldn't have classes in Excel and Powerpoint, and we wouldn't have classes in SPSS and SAS, and we wouldn't even have classes in Python -- we'd have classes in envars, registry, and shell scripts/bat files.

On Fri, 26 Feb 2021 at 09:07, Steven D'Aprano steve@pearwood.info wrote:

On Fri, Feb 26, 2021 at 11:41:56AM +0900, Stephen J. Turnbull wrote:

...

Mike Miller writes:

...

"sys-admin" is a bit of an overstatement in my phrasing. The core is that you need to understand how a PATH works and be able to run pip and cd into folders and perhaps run rm/del, etc. Basic command-line skills?

That's what I would mean by basic sys-admin skills. And *surprise!* my students don't have them, and don't need them ... until they start using Python.

Is it *only* Python though? Wouldn't that be necessary if they were learning Perl, Java, C, Ruby etc?

It is possible to teach students to do most things in Python without using the command line. For example you can tell them all to install anaconda and then use spyder as their editor. Or you can get them to install and use vscode or pycharm or some other IDE with built in virtualenv and git integration etc.

Where Python is awkward is at the point where you actually *want* to teach students to use the command line. The problem is that Python is not set up for command line use out of the box in a consistent way across different platforms.

It would be nice to say: now open a terminal, type "python", and hit the enter key and you'll see the Python prompt >>> but it's just not that simple so the instructions end up being like: """ It might be possible to run python from the terminal by typing "python". However that might not work or it might run the wrong version/installation of Python. It might be that you haven't added Python to your PATH environment variable or maybe you have depending on which boxes you ticked when installing. Possibly you need to change your PATH environment variable or possibly you should run Python as "python3" or "py" instead but I have no way of saying without knowing what OS you are on, where you installed Python from, what options you selected in the installer, whether you have any other Python installations etc. If you are on Windows and you get "permission denied" then that's possibly because the MS-provided python.exe stub is on your PATH so you first need to disable that in "app execution aliases". Possibly you can't run Python from the normal terminal but you can open a special terminal for Python that does have the PATH set up such as the Anaconda Prompt.

Okay next lesson is how to use "pip" which is really easy: just type "pip install X" but maybe you should actually type "pip3" or "python -m pip" or "python3 -m pip" or "py -m pip" or actually you if you installed Anaconda you might need to use "conda install X". """ When someone learns to use the command line for the first time it's really quite helpful if you can tell them exactly what commands to type. Knowing how to configure PATH and understanding what it does and understanding about OS differences etc is potentially useful in the long run but is a major distraction at the beginning.

There have been efforts to improve the situation but in some ways they make it worse. For example the py launcher supposedly fixes some of these problems on Windows but the fact that there is no analogue of it on any non-Windows platform means that any instructions involving it immediately need to split according to OS. Likewise the new python.exe stub that ships with Windows is supposed to fix the Python PATH problem but it actually *prevents* python from working in the terminal if you have installed Python some other way (even if you have added it to PATH): https://stackoverflow.com/questions/56974927/permission-denied-trying-to-run...

The problems aren't only on Windows because you also have "python" vs "python3" on OSX, Linux etc. On OSX you can also have conflicts between the system Python and any user installed Python. On OSX some installers set up the PATH for you by automagically editing .bash_profile which can be helpful but can also be confusing. After some time people don't actually know how to run the different versions of Python that they installed through homebrew, from python.org or from Anaconda etc and their .bash_profile is a mess. The question of exactly what you should type to run python, pip, etc applies on Linux although I find it less problematic just because Linux users typically know how to use the terminal already.

For me as a user of Python I don't usually notice these problems because I use some sort of virtual environment for everything so I can always type "python" or "pip" and there are no conflicts between different installations. When a student can't even get "python" to work in the terminal though it's too much to ask them to start setting up virtual environments.

-- Oscar

I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.

The problem is made worse because it often hits at the beginning instead of 7 lessons in, when they would have some reason to think it is worth struggling through and will eventually work.

On Sat, Feb 27, 2021 at 01:04:08AM +0000, Oscar Benjamin wrote:

On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett jimjjewett@gmail.com wrote:

...

...

I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.

Thanks Jim, that is the situation and yes they are not CS students.

Why is that relevant?

I'm not an automotive engineer, I haven't studied for an Automotive Engineering Technology (AET) bachelor degree, but I still need to know how to change a flat tire, fill the petrol tank with fuel, keep the windscreen wiper water tank filled, put snow chains on if I live in a region with snow, etc. Or pay somebody else to do it.

Python is not Scratch. It doesn't, and shouldn't, aim to be a self-contained environment that isolates the user from their own computer.

[Aside: the Scratch interpreter will actually refuse to run if it thinks your code or documentation includes "bad words":

https://i.redd.it/nyqra8lmbei21.png

https://scratch.mit.edu/discuss/topic/32001/

https://scratch.mit.edu/discuss/topic/122410/

If this doesn't make you want to take off and nuke the entire planet from orbit, then I don't know what's wrong with you. End of rant.]

It doesn't matter whether you are using Windows, Mac OS X or Linux, whether your preferred UI is the command line or point-and-click, there are some basic fundemental skills that every amateur or professional programmer needs to know, such as:

- how to edit files and save them

- which file am I actually running?

- which interpreter am I actually running?

- how do I tell the computer to use a different interpreter?

These should be basic pre-requisites before learning to code, like being able to read and write.

[...]

I see that the official Python tutorial now suggests typing "python3.9" [1]. Is that what is recommended now? Obviously that would fail for someone who had installed 3.8.

It isn't elitist to expect that, at the barest minimum, any student of programming should be able to adjust the command

python3.9 hello.py

to

python3.8 myscript.py

as required. If you can't debug

python3.9: command not found

or

can't open file '/home/steve/hello.py': [Errno 2] No such file or directory

then you're probably going to have a miserable time dealing with *hard* bugs. Other more subtle errors may require a bit more knowledge, but that's okay. We don't need to make programming a knowledge-free activity.

The other point though is that it doesn't need to be like this. If the issue was just installing Python and then setting up your PATH then that's manageable. The problem is that even after doing those things there isn't a "one way" to invoke Python from the command line. All possible invocations (python, python3, py, ...) will fail for some users. That's a problem for beginners but it's also a problem in any situation where you want to write a command line that should run on multiple platforms (e.g. in a Makefile or some other kind of script).

Writing code to run on any platform is a hard problem that requires complex solutions. If you need to write a makefile that will run on a million different flavours of Unix, including rare, old and weird ones, then you have to expect that's a hard problem and you're probably going to need to spend some effort getting it to work.

Not every problem is capable of being solved by a beginner with no knowledge, and we don't have to bust our guts to make it an easy problem. It's okay to delegate some problems to someone else, such as make experts.

It would be great if Python could converge around a single way for users to invoke the Python that they have installed ... py also has features for selecting different versions

I don't think that "py --someoption ..." is any better than the existing solutions I already have for selecting different versions. Maybe on Windows? I don't know how it works there.

Jim J. Jewett writes:

...

which file am I actually running? which interpreter am I actually running? how do I tell the computer to use a different interpreter?

If you need to care about any of these, then the environment is fighting you -- and the application probably stinks. Programmers have to deal with it because of bootstrapping, but there is no reason that we should assume all python users need that flexibility or want that responsibility.

Be careful! I agree about need for flexibility and desire for responsibility, but in fact these problems present themselves frequently when helping people with their work. "Which file am I actually [edit]ing?" is one that comes up in just about every meeting: documents are undated (or dated with the deadline date), people are looking at different versions, and not infrequently the presenter themself has a Documents/Downloads confusion and gets it wrong. Then "where is the one I want to be editing?" requires an oracle. :-) "Which program am I running?" doesn't come up that often, but it does come up when helping people with "doing something", especially trying to find menu items across major versions. Interestingly enough, "how do I tell the computer to use a different interpreter?" doesn't come up for me outside of Python (and even there rarely) -- everybody who consults me knows how to use "Open With".

So I agree with the basic principle that programs should Just Work by default, but it's not always possible. Eg, there's good reason why Downloads and Documents both exist. So it helps to have users know something about implementations and sysadminning stuff, especially when remote troubleshooting.

The question for Python is given the history and the plethora of ways to invoke Python and find packages and programs, can we do better with a minimum of backward incompatibility? Humans are generally better at learning this stuff than bags of bits on spinning platters are!

On 2/28/21, Oscar Benjamin oscar.j.benjamin@gmail.com wrote:

• It is possible to configure a default version (although I think you

have to do it with an environment variable)

The py launcher in Windows supports a "py.ini" file beside the executable and in %LocalAppData%. The equivalent of the PY_PYTHON, PY_PYTHON2, and PY_PYTHON3 environment variables can be set in the "[defaults]" section as "python", "python2", and "python3" settings.

The ini file also supports a "[commands]" section to define additional virtual commands for shebangs. Regular filepaths are also supported in shebangs -- e.g. #!"path\to\any\program.exe".

• I think that the launcher is only installed in an all users install.

It defaults to an all-users install, but it can also be installed for just the current user in "%LocalAppData%\Programs\Python\Launcher". In this case, the installation directory always gets added to PATH.

• Listing installations with "py -0p" is somewhat cryptic

There's also the long-form options `--list` and `--list-paths`.

• It would be better if you could use the launcher itself to set the

default Python e.g. "py --set-default-python=3.8"

It's pretty simply to run `set PY_PYTHON=3.8`, and persist the value in the registry with `setx.exe PY_PYTHON 3.8`. (But don't use setx.exe naively to set PATH.)

On the last point I think that although Anaconda doesn't install the launcher you can use the launcher to run the python from the Anaconda installation.

I don't use Anaconda, but I don't think that's supposed to be the case according to PEP 514. The launcher only looks for PSF development distributions in the "PythonCore" registry key.

On 2/28/21, Oscar Benjamin oscar.j.benjamin@gmail.com wrote:

Oh, okay. So does that mean that it's always on PATH unless the user *explicitly unticks* the "install the launcher" box for both single user and all user installs?

If the launcher gets installed, it will be available in PATH. IIRC, the installer only allows installing the launcher for the current user if it is not installed already for all users. Thus only one version should ever exist in PATH, which is set in either the user or system "PATH" value in the registry, but not both.

On Sat, 27 Feb 2021 at 11:04, Paul Moore p.f.moore@gmail.com wrote:

On Sat, 27 Feb 2021 at 01:08, Oscar Benjamin oscar.j.benjamin@gmail.com wrote:

...

The other point though is that it doesn't need to be like this. If the issue was just installing Python and then setting up your PATH then that's manageable. The problem is that even after doing those things there isn't a "one way" to invoke Python from the command line. All possible invocations (python, python3, py, ...) will fail for some users. That's a problem for beginners but it's also a problem in any situation where you want to write a command line that should run on multiple platforms (e.g. in a Makefile or some other kind of script).

This is 100% true. (At least the part where you say that "it's a problem". I'm not sure if "it doesn't need to be like this" is true - can you point to a language where the command line "just works" the way you suggest?)

Python from 15 years ago :)

Maybe I'm misremembering but when I first started with Python there didn't seem to be any ambiguity about what to type in the terminal. The first split as I remember it was on Linux when python3 came along but python kept referring to 2.x on (most) Linux. At the same time python3 was not added to Windows and instead python became 3.x. Then there was the py launcher for Windows. Perhaps this problem always existed on OSX although I didn't use it at all at the time.

More recently I've been exploring Julia. You could compare these pages:

https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter https://docs.julialang.org/en/v1/manual/getting-started/

When installing julia you select the installer for your OS and there are some instructions for setting up PATH (although I didn't need to do that manually on OSX). After setup the instruction is that you type "julia" in the terminal and then Ctrl-D to exit. There are no caveats or mentions of different operating systems or words like "if" and "usually". The setup process is platform specific but then the usage instructions are not.

-- Oscar

On Sat, 27 Feb 2021 at 13:03, Oscar Benjamin oscar.j.benjamin@gmail.com wrote:

More recently I've been exploring Julia. You could compare these pages:

https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter https://docs.julialang.org/en/v1/manual/getting-started/

When installing julia you select the installer for your OS and there are some instructions for setting up PATH (although I didn't need to do that manually on OSX). After setup the instruction is that you type "julia" in the terminal and then Ctrl-D to exit. There are no caveats or mentions of different operating systems or words like "if" and "usually". The setup process is platform specific but then the usage instructions are not.

Sorry, I couldn't resist replying to this.

I went to the Julia page. That linked me to the download page. Which offered me multiple versions. Which should I take? Stable, LTS or "upcoming"? As a beginner without the experience I have hitting stuff like this, how would I know? Let's assume "stable". OK, download and run an installer is easy enough for me on Windows, but let's say I'm doing this on Linux, I get the choice of "Generic Linux on x86" or "Generic Linux on ARM". Huh? I'm on Ubuntu, not "Generic", so I'm confused. Then I have "64-bit (GPG)" or "64-bit (musl) (GPG)" Or AArch if I chose "ARM". How the heck would I know. Any of those get me a .tar.gz file, which I don't know how to use...

Going to the Linux help, I get told to use wget, not the file I just downloaded. And then I unpack it, and now I'm into setting PATH to /tmp/julia-something/bin (because I did this in /tmp, as I have no idea where I want to put this long term).

OK, my linux thought experiment isn't going well. Let's go back to Windows, where I was lucky enough to pick "installer" rather than "portable", so I avoided a world of pain about where to put the package, just like I hit on Linux. The installer puts Julia in my personal apps directory, just like Python does. I'm going to do something "advanced" here ;-) and deselect "desktop shortcut" because I hate clutter on my desktop. OK, it's complete, let's select "run julia". That got me a prompt. Cool. Start menu item does the same. Also cool.

BUT... The python installer on windows is *exactly the same*. Installs to user's area, start menu entry that runs the REPL. So that's no worse.

Suppose I go to a command prompt on my own and try to run Julia. Open a terminal, "julia". No good, I need to set PATH. Hmm, wasn't there something in the help on the Julia website about that? Off we go to check. Weren't we here before when we were using Python? And the instructions on the Julia website are about as complex as I'd expect, and much the same as for Python. So still no worse for Python. But the Python installer has "add Python to PATH" that does this stuff for you (not selected by default, because of reasons that you may or may not agree with, but at least it's *there*). And there's the py launcher, but let's ignore that, as it works round some complexity at the cost of some different complexity, so it's a distraction at this stage.

Honestly, I think Julia is overall slightly *worse* than Python (at least on Windows). But it's almost identical in practice.

The one thing that *is* substantially worse for Python, is the circumlocutions needed in the documentation to say how to run Python. But that's 100% down to us not being willing to say "just type the command python". And the reason for *that* is mostly historical, related to the Python 3 transition and what happened on Linux over the python/python2/python3 commands, and to a lesser extent the introduction of the launcher on Windows ("lesser", because using the launcher on the command line wasn't recommended until some time after its introduction, at which point the python2/python3 split was established)...

Paul

Executive summary: This thread is about a "seven blind men and an elephant" problem.

Jim J. Jewett writes:

I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.

Not sure if "his" refers to me, but I don't know what they're going to need to use. C/C++, maybe not, but there are a lot of Perl and especially Java shops in Tokyo still. The largest group of my students end up as what the Japanese call "SE" ("systems engineers"), by which they mean a sales/requirements gathering/training/customer relations role. Many of the rest end up in tech-adjacent roles. I think all of them would benefit from a little command-line experience, and a better understanding of how a group of programs cooperates to become a system. As their teacher, I certainly would!

The problem is made worse because it often hits at the beginning instead of 7 lessons in, when they would have some reason to think it is worth struggling through and will eventually work.

I think we're on the same page, but "7 lessons in" is still early in my experience teaching Python. Languages where you have to explicitly invoke a compiler and then run the executable, maybe you run into the problem "at the beginning" (shades of Chapter 1 of _The C Programming Language_!), but not with Python. Where my students run into problems is when they *know* it was all worth it, because *now* they're ready to use Python to do whatever it is they style "real work" -- and need to install, maybe configure, and learn to use something that isn't an included battery. Even in a well-curated framework like Anaconda, "you can make a bottle adult-proof, but not child-proof".

So *my* point (I don't speak for anyone else) is pessimistic. I don't think there's a one-size-fits-all solution to child-proofness. What we need is children with better vocabularies.

From my point of view, what I want is to talk to students (and colleagues!) about problems that I can often solve or install a workaround in a few minutes. But when simple questions like "what distribution did you install?" gets the answer "I just downloaded the package. What's a distribution?", and the followup is, "well, where did you get it?", "I clicked on the top link in Google", which leads to "well, is it this one?" and "I can't tell...", it's very frustrating for all concerned -- and we haven't even started on the new package that isn't working.

When to them, it's pure fsckin' magic, and to me, it's muscle memory, problem-solving across the gap is hard. For my purpose, cross- platform distributions like Anaconda seem to be the best solution available now and for the near future. But for many developers, they have enough problems without dealing with the idea that some other application has different version or configuration requirements for some components.

I just don't see a world where any of the points of view are without merit. People with more space than apps can put them *all* in venvs, people with relatively simple "external" requirements can just use a single installation, and people with the necessary skills can mostly use the single installation and put a few "problem children" into venvs of their own. Each of those groups will want Python itself to take a different approach to the "problem".

So the question in the subject is valid and should be raised once in a while, but the answer is always going to disappoint a majority. :-)

Steve

On Wed, 24 Feb 2021 at 10:18, Antoine Pitrou antoine@python.org wrote:

On Tue, 23 Feb 2021 20:29:52 -0500 Jonathan Goble jcgoble3@gmail.com wrote:

...

I can't speak for distributors or maintainers [1], but I can speak for myself as a user. I run Debian testing (currently bullseye as that is preparing for release) as my daily OS on my personal laptop, used for personal matters and school assignments (I'm a university computer science student in my senior year).

I don't use the system Python for anything of my own, whether it's a school assignment or a personal project, precisely because I don't want to risk screwing something up. Rather, I maintain a clone/fork of the official CPython GitHub repo, and periodically build from source and `make altinstall` into `~/.local/`.

For the record, instead of building Python by hand, you could use a distribution such as Anaconda or the community-maintained conda-forge.

Regards

Antoine.

I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been quite happy with that, and it basically does what Jonathan does manually: clone the github repo and build python from scratch.

FWIW, Distro repacks are advantageous in comparison to "statically-bundled" releases that for example bundle in an outdated version of OpenSSL, because when you `apt-get upgrade -y` that should upgrade the OpenSSL that all the other distro packages depend upon.

Here's something that doesn't get called as frequently as `apt-get upgrade -y`:

```bash pip install -U certifi ```

https://github.com/certifi/python-certifi (Mozilla's CA bundle extracted into a Python package)

```bash apt-get install -y ca-certificates dnf install -y ca-certificates ```

On 2/24/21, Wes Turner wes.turner@gmail.com wrote:

On 2/23/21, Random832 random832@fastmail.com wrote:

...

I was reading a discussion thread https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:

Christian Heimes wrote:

...

Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.

I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?

I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.

How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?

First, pip+venv is not sufficient for secure software deployment: something must set appropriate permissions so that the application cannot overwrite itself and other core libraries (in order to eliminate W^X violations (which e.g. Android is solving by requiring all installed binaries to come from an APK otherwise they won't and can't be labeled with the SELinux extended file atrributes necessary for a binary to execute; but we don't have binaries, we have an interpreter and arbitrary hopefully-signed somewhere source code, at least)).

Believe it or not, this is wrong:

# python -m venv httpbin || virtualenv httpbin
# source httpbin/bin/activate
mkvirtualenv httpbin

pip install httpbin gunicorn
gunicorn -b 0.0.0.0:8001 httpbin:app

# python -m webbrowser http://127.0.0.1:8001

It's wrong - it's insecure - because the user executing the Python interpreter (through gunicorn, in this case) can overwrite the app. W^X: has both write and execute permissions. What would be better?

This would be better because pip isn't running setup.py as root (with non-wheels) and httpbin_exec can't modify the app interpreter or the code it loads at runtime:

useradd httpbin # also creates a group also named 'httpbin'
sudo -u httpbin sh -c ' \
    python -m venv httpbin; \
    umask 0022; \
    ./httpbin/bin/python -m pip install httpbin gunicorn'

useradd httpbin_exec -G httpbin
sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app'

This would be better if it worked, though there are a few caveats:

sudo apt-get install python-gunicorn python-httpbin
sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app

1. Development is impossible:

• You can't edit the code in /usr/lib/python3.n/site-package/ without

root permissions.

• You should not be running an editor as root.
• You can edit distro-package files individually with e.g. sudoedit

(and then the GPG-signed package file checksums will fail when you run `debsums` or `rpm -Va` because you've edited the file and that's changed the hash).

• Non-root users cannot install python packages without having someone

repack (and sign it) for them.

• What do I need to do in order to patch the distro's signed repack of

the Python package released to PyPI?

• I like how Fedora pkgs and conda-forge have per-package git repos now.
• Conda-forge has a bot that watches PyPI for new releases and tries

sending an automated PR.

• If I send a PR to the main branch of the source repo and it gets

merged, how long will it be before there's a distro repack built and uploaded to the distro package index?

2. It should be installed in a chroot/jail/zone/container/context/vm

so that it cannot read other data on the machine. The httpbin app does not need read access to /etc/shadow, for example. Distro package installs are not - either - sandboxed.

To pick on httpbin a bit more, the httpbin docs specify that httpbin should be run as a docker container:

docker run -p 80:8001 kennethreitz/httpbin

Is that good enough? We don't know, we haven't reviewed:

• the Dockerfile
  • it says `FROM ubuntu:18.04`, which is fortunately an LTS release.

But if it hasn't been updated this month, it probably has the sudo bug that enabled escalation to root (which - even in a container - is bad because it could obnoxiously just overwrite libc, for example, and unless the container is rebuilt or something runs `debsums`, nothing will detect that data integrity error)

• the requirements.txt / setup.py:install_requires / Pipfile[.lock]

dependencies

• does it depend upon outdated pinned exact versions?
  • Is there an SBOM (Software Bill of Materials) that we can review

against known vulnerability databases?

How do I know that:

• The packages I have installed are not outdated and unpatched against

known vulnerabilities

• The files on disk are exactly what should be in the package
• The app_exec user can't overwrite the binary interpreter or the

source files it loads at runtime

• There won't be unreviewed code running as root (including at install

time)

• All Python package dependencies are available as wheels (that

basically only need to be unzipped)

• The ensemble of dependencies which I've miraculously assembled is

available on the target platform(s)

• The integration tests for my app pass with each combination of

dependencies which satisfy the specified dependency constraints

• I can edit things and quickly re-test
• Each dependency is signed by a key that's valid for that dependency

So, if pip is insufficient for secure software deployment, what are pro teams using to build signed, deployable artifacts with fresh, upgaded dependencies either bundled in or loosely-referenced?

• Bazel (from Google's internal Blaze) builds from BUILD files.
  • https://github.com/dropbox/dbx_build_tools
• Pantsbuild, Buck
• zipapps
• FPM can apparently package up an entire virtualenv; though IDK how

good it is at permissions?

https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/virtualenv.r...

As an open source maintainer, there are very many potential environments to release builds for. Manylinux docker images (and auditwheel, delocate, and *cibuildwheel*) are a response to extreme and somewhat-avoidable complexity. https://github.com/joerick/cibuildwheel

Distro packagers can and do build upon e.g. pip; which is great for development but not sufficient for production deployment due to lack of support for file permissions, extended file attributes, checksums, cryptographic signatures, and due to running setup.py as the install user for non-wheel packages.

There are many deployment stories now: pull/push, configuration management systems, venvs within containers within VMs. For your favorite distro, how do I get from cibuildwheel to a signed release artifact in your package index; and which keys can sign for what?

On 24/02/2021 08.03, Michał Górny wrote:

On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:

...

I was reading a discussion thread https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:

Christian Heimes wrote:

...

Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.

I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?

I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.

The problem is a little deeper and the Debian solution doesn't really solve all of it.

Yes, pip installing into the same directory as the package manager is a problem. It's a problem to the point that I'm patching pip in Gentoo to explicitly block that. We've gotten far too many bug reports about people's systems suddenly being horribly broken after they used pip.

While using two directories can prevent pip from directly overwriting system packages, you still can't expect two independent package managers to work simultaneously unless they can communicate with each other to prevent conflicts. If pip installs a different version of the same package as the package manager, which one is supposed to be used? Whichever choice you make, you'll bound to eventually break dependency graph of some package.

Thanks Michał, you have explained the situation well.

Most Linux distros have two separate directories for distro packages and "sudo pip install" packages. This is a great thing!

Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system. The package manager of several Linux distributions is written in Python. If your system Python is broken, your package manager is likely busted as well. The split directories allow ysers to fix their system with a single command like "sudo rm -rf /usr/local/lib/python*" (paths may vary).

The issue is: Split distro/pip directories are not baked into Python or pip. Distros have their own set of downstream patches that modify the behavior of Python. Some of the patches have caused issues in the past.

I would like to see this feature implemented in Python's core, so distro packages have to do less patching and upstream pip can test against it.

Christian

PS: I wrote PEP 370 for "install --user" 13 years ago before we had virtual envs.

On 24/02/2021 15.16, Random832 wrote:

On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:

...

Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.

I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.

Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installation.

Yes, you are describing one major aspect of my idea for a system Python interpreter. I'm happy to read that other users are coming to similar conclusions. Instead of an option I'd create a new executable to lock down additional things (e.g. isolated mode, code verification hook). A separate executable would also allow distros to provide a stripped down interpreter that does not cause bad user experience.

On Wed, 24 Feb 2021 at 18:25, Steve Dower steve.dower@python.org wrote:

I mean, this is _precisely_ what PEP 370 defines (including the "-s" option and PYTHONNOUSERSITE env variable to provide that one-way isolation).

Is the problem that pip doesn't use it by default?

Note that these days, pip *does* use usersite by default if site-packages isn't writable. Of course, if users run pip under sudo, that invalidates that mechanism :-(

Paul

On 24/02/2021 20.03, Christian Heimes wrote:

On 24/02/2021 19.17, Steve Dower wrote:

...

On 2/24/2021 4:26 PM, Christian Heimes wrote:

...

On 24/02/2021 15.16, Random832 wrote:

...

On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:

...

Separate directories don't prevent clashes and system breakage. But they provide an easy way to *recover* from a broken system.

I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.

Basically, site-packages would effectively be layered on top of "Lib

• dist-packages" in a similar way to how a venv is layered on top of

the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installati

on.

...

Yes, you are describing one major aspect of my idea for a system Python interpreter. I'm happy to read that other users are coming to similar conclusions. Instead of an option I'd create a new executable to lock down additional things (e.g. isolated mode, code verification hook). A separate executable would also allow distros to provide a stripped down interpreter that does not cause bad user experience.

I mean, this is _precisely_ what PEP 370 defines (including the "-s" option and PYTHONNOUSERSITE env variable to provide that one-way isolation).

Is the problem that pip doesn't use it by default? Or that the distros went and made patches for the runtime rather than patching pip? (For Windows installs from the Store, where even admin rights can't do an all-users package install, I added a new config file location for pip to change this default, but a patch would also have worked.)

Maybe we need an easier way to patch the location of user site packages? I also had to do this for the Store install on Windows, and it's a little bit of a hack... but maybe having an official recommendation would help encourage distributors to use the mechanism?

(sorry for terse reply, I'm heading out.)

Linux distros want an additional layer so the can differentiate between Python packages installed by their package manager and Python packages installed with "sudo pip install".

Python has two default site-package directories (from highest to lowest precedence):

1. ~/.local/lib/python3.9/site-packages
2. /usr/lib/python3.9/site-packages

(1) is defined by site.getusersitepackages(), (2) site.getsitepackages()

Linux distro have additional site-package directories. My Fedora system X86_64 system with multiarch has three additional candidates.

1. ~/.local/lib/python3.9/site-packages
2. /usr/local/lib64/python3.9/site-packages
3. /usr/local/lib/python3.9/site-packages
4. /usr/lib64/python3.9/site-packages
5. /usr/lib/python3.9/site-packages

The "lib" directories are for pure Python packages whiel the "lib64" directories contain X86_64 native code extensions (aka shared libraries). The "/usr/lib*" directories are used for distro packages. The downstream patch [1] ensures that "sudo pip install" installs packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.

Debian has a similar mechanism to provide "/usr/lib/python3/dist-packages" [2].

A hypothetical /usr/bin/system-python3 interpreter would only use (4) and (5) as site-package directories. A user-facing /usr/bin/python3 would use (1) to (5).

Christian

[1] https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-use... [2] https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

PS: Fedora patch [1] uses -s / PYTHONNOUSERSITE to exclude "/usr/local/lib*" site-package directories.

$ python3 -c 'import site; print(site.getsitepackages())' ['/usr/local/lib64/python3.9/site-packages', '/usr/local/lib/python3.9/site-packages', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']

$ python3 -s -c 'import site; print(site.getsitepackages())' ['/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']

Is there a tool that (1) detects import name collisions; and (2) attempts to read package metadata and package file checksums (maybe from the ZIP 'manifest')?

In order to:

- troubleshoot module shadowing issues - $PATH - sys.path - `python -m site` - incomplete and overlapping uninstallations:

pip install a pip install a_modified # pip uninstall a? pip install pdbpp pip uninstall a_modified ls -altr "${site-packages[*]}" strace -e trace=file python -c 'import pdb'

****

When shouldn't site customizations be added to the site module? https://docs.python.org/3/library/site.html

When should customizations be built into the build instead of a runtime conditional?

On Sat, Feb 27, 2021, 23:12 Nick Coghlan ncoghlan@gmail.com wrote:

On Wed, 24 Feb 2021 at 10:49, Random832 random832@fastmail.com wrote:

...

I was reading a discussion thread <

https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e%3E about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:

...

Christian Heimes wrote:

...

Core dev and PyPA has spent a lot of effort in promoting venv because

we don't want users to break their operating system with sudo pip install.

...

I don't think sudo pip install should break the operating system. And I

think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?

The reason venv is promoted as heavily as it is is because it's the only advice that can be given that is consistently correct regardless of the operating system the user is running locally, whereas safely using a system-wide Python installation varies a lot depending on whether you're on Windows, Mac OS X, or Linux (let alone some other platform outside the big 3 desktop clients).

conda is also popular for the same reason: while the instructions for installing conda in the first place are OS-dependent, once it is up and running you can use consistent platform independent conda commands rather than having to caveat all your documentation with platform-specific instructions.

Apple moved all of their dynamic language interpreter implementations to inaccessible-by-default locations so Mac OS X users would stop using them to run their own code.

Alongside that, we *have* worked with the Linux distro vendors to help make "sudo pip install" safe (e.g [1]), but that only helps if a user is running a new enough version of a distro that has participated in that work.

However, while the option of running "platform native" environments will never go away, and work will continue to make it less error prone, the level of knowledge of your specific OS's idiosyncrasies that it requires is almost certainly going to remain too high for it to ever again become the default recommendation that it used to be.

Cheers, Nick.

[1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note: this change mitigated some aspects of the problem in a way similar to what Debian does, but still doesn't solve it completely, as custom Python builds may still make arbitrary changes)

P.S. "But what about user site-packages?" you ask. Until relatively recently, Debian didn't put the user's local bin directory on the system path by default, so commands provided by user level package installs didn't work without the user adjusting their PATH. The CPython Windows installer also doesn't adjust PATH by default (for good reasons). And unlike a venv, "python -m" doesn't let you ensure that the code executed is the version installed in user site-packages

• it could be coming from a directory earlier in sys.path.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RDLEH6DU... Code of Conduct: http://python.org/psf/codeofconduct/