[Python-ideas] Looking for input to help with the pip situation

[Nick Coghlan]

On 14 November 2017 at 07:52, Paul Moore <p.f.moore at gmail.com> wrote:
> On 13 November 2017 at 18:57, Chris Barker <chris.barker at noaa.gov> wrote:
>> This has gotten to be a big thread, and it's a pretty intractable problem,
>> but I think there are a few fairly small things that could be done to at
>> least make it a bit easier:
>
> In principle, I agree with the ideas here, but there are some
> practical issues that make them somewhat less straightforward than we
> might like.
>
>> 1) Add python2.exe and python3.exe files to the Windows installers -- am I
>> insane or did Windows used to have that? I really think it did -- maybe got
>> removed when py.exe was added.
>
> I don't think Windows ever had python2.exe/python3.exe, but I could be wrong.

Not that I'm aware of in the python.org installers, and I don't think
ActivePython does either. I'm less sure about Enthought or Anaconda
(since I've never used either of them on Windows).

> The only possible way we'd get a python2.exe is by adding it to future
> releases of Python 2.7, and we shouldn't be recommending Python 2 for
> new users anyway. And I'm strongly -1 on promoting "python3.exe" as
> the canonical way of getting Python 3. The python3 command is a result
> of the Unix issues around switching to Python 3 as the default, and we
> shouldn't perpetuate that approach on Windows, where it's unneeded.

Right, we actually want to get to a state where "python" as a symlink
to "python3" is a supported system configuration, but we need to go
through a period of it not linking to anything at all first (to ensure
that anyone that currently needs Python 2 has a chance to either
switch to explicitly calling python2 instead, or else update their
software to be Python 3 compatible).

While the specifics are out of date (due to changes we've made in the
past few weeks while working on the F27 Modular Server release),
https://fedora-python.readthedocs.io/en/latest/plans/default-python-module/
sketches out a possible approach to getting Fedora there by the end of
2018 (in Fedora 29, as part of the switch to CPython 3.7).

The other reason we don't want to permanently entrench `python3` as a
preferred command is that we assume that Python 4.x will eventually
happen in some form or another, and we'd like to get to a point where
`pythonX` is how you request a particular *old* version of Python,
with the unqualified `python` typically meaning the latest available
version.

> Having said that, I don't object to including versioned executables as
> a convenience (if it is one). I just dislike promoting the use of them
> via standard documentation.
>
>>   1a) alternatively, we could add a "py" executable to the standard linux
>> builds, so there would be THAT one way to do it. But I think that's a "BAD
>> IDEA" -- the whole "py" thing is not widely know or used, it's not going to
>> show up in package install instructions for a LONG time, (actualy we could
>> do both anyway)
>
> I think that getting a "py" launcher on Unix is a lost cause at this
> stage. It would be nice, and maybe if the original PEP had proposed a
> cross-platform py command, it might have worked, but that's history
> now, and I think the py launcher will probably always be a
> Windows-only thing.

Yeah, myself, Barry Warsaw, Matthias Klose, and a number of other
folks on linux-sig have poked at this idea multiple times since Geoff
Thomas first posted about https://github.com/geofft/pythonmux, but our
conclusion each time has been that it wouldn't help enough to justify
the effort involved in implementing and promoting it. After 20+ years
of usage in the Linux ecosystem, `/usr/bin/python` and `/usr/bin/env
python` are simply too entrenched in both people's habits and existing
code.

Hence the approach the mainstream distros have ended up taking: break
references to `python` in our default installs, offer a
straightforward way to get that back as a reference to Python 2.7, and
then start trying to figure out a supportable mechanism that will
allow system operators to opt in to having it point to Python 3.x
instead. Ultimately, this isn't that different from what Arch did back
in 2011, just much later in the 3.x adoption cycle, and with a simpler
and more obvious forced breakage (a missing command), rather than the
potentially cryptic errors emitted when attempting to run Python 2
code on Python 3.

A couple of other points worth noting here:

* https://github.com/python/redistributor-guide/issues/1 which links
to https://github.com/python/peps/pull/315/files, which was a draft
replacement for PEP 394 that we never actually merged. We know
external visibility into these distro-level discussions is currently
poor, and doing something about that is on the todo list, we(/I) just
haven't gotten to it yet
* even on Windows, the main intended purpose of the `py` launcher was
to more reliably handle file associations by introducing an equivalent
to POSIX shebang line processing. The interactive command line use
case was more of an added bonus than anything else.

>> Then "python2 -m pip install" would work everywhere (only with new
>> installations, but at least with newbies, that's a bit more likely ...)
>
> No newcomer should *ever* be getting told to use "python2 -m pip
> install". Maybe "python3 -m pip install", but never Python 2 (outside
> of specialised environments where training people in an out of date
> version is required). And of course not all Unix distributions come
> with Python 3 installed (Mac OS being an obvious example, and I think
> Nick mentioned CentOS and RHEL) so python3 won't work everywhere
> either...

Right, getting a regular Fedora-style python3 install on RHEL/CentOS
currently requires enabling the IUS community repos, as per the last
option in https://packaging.python.org/guides/installing-using-linux-tools/#centos-rhel

We do offer python3-by-default as part of OpenShift, though, where the
base container images auto-activate the Python 3 Software Collection,
such that `python` refers to Python 3.x by default, just as it does
inside a Python level virtual environment:
https://github.com/sclorg/s2i-python-container


> The only really sensible "do what I expect" situation is a single
> Python 3 installation on the user's machine. For that case, adding
> Python to PATH (it doesn't matter where as there's only one) is a
> sensible option. The downside is that optimising for that case makes
> the behaviour for other (more experienced) users worse. But it's not
> unreasonable to make that trade-off.

Yep, I think add CPython to the user PATH by default will still be helpful:

1. If there's no system Python, then it should "just work"(tm)
2. If there is a system Python, and the user has admin privileges,
then "Take these entries from your user PATH and add them to the
system PATH" is a better trouble-shooting step
3. Otherwise, they're in the same situation as a Linux user wanting to
run a non-default Python, and the solution is similar (i.e. learn to
dynamically manipulate the environment, perhaps with the aid of a tool
like conda)
4. The Windows installers are one of the cases where python.org *does*
get to quickly alter typical new user behaviour, since we control the
download links

(Related: if anyone wants to tackle a potentially interesting web UX
design problem, right now, Python 3.6.3 and 2.7.14 are presented as
"peer" downloads on python.org, with no obvious guidance on choosing
between them. This should probably be separated out, such that folks
that already know they need 2.7 have no trouble finding it, but 3.6 is
clearly presented as the default choice for folks that don't already
know which version they want)

>> 3) Make --user be be automatic for pip install. Not actually the default,
>> but pip could do a user install if you don't have the permissions for a
>> non-user install.
>
> The problem here is that the user scripts directory isn't on PATH. We
> could put it on, but again we hit the "goes after the system PATH"
> problem (on Windows).

We should still optimise the defaults for the desired system
configuration (i.e. user-mode installs as the cross-platform default
outside a venv), and then work on platform-specific troubleshooting
guides for the common ways that things can go wrong.

> But in principle, if the issues can be solved in a way
> we're all happy with, I agree with the principle that "download and
> install Python" should result in an environment where
>
>     python myscript.py
>     pip install something_or_other
>
> "just work" if the user previously had no Python on their PC. Anything
> less results in a worse beginner experience for Python than for other
> languages (which usually don't support having multiple versions
> installed at once, and often require admin installs, so bypass most of
> the problems Python has to deal with).

As per https://github.com/pypa/python-packaging-user-guide/issues/396#issuecomment-343739110
and Chris's example of "Analyse your Python environment" as a
student's first script, we may be able to get to a point where we can
say:

1. Try these commands. If they work, excellent, your Python
environment is correctly set up, and there's nothing further to be
done.
2. If they don't, then either:
- follow [platform specific troubleshooting guide]; or
- follow [platform specific venv bootstrapping guide]

With any luck, a Python installation troubleshooting guide with
concrete steps to try may also prove more popular as a third party
link target than installation instructions that assume you're starting
with a clean system :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia