[Distutils] Versioned entry points

[Nick Coghlan]

On 29 March 2014 07:23, Paul Moore <p.f.moore at gmail.com> wrote:
> On 28 March 2014 21:06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> So consider me in the school that suggests this be a standard installer
>> feature that can be applied to any entry point script, rather than something
>> that varies by project.
>
> Doing that also implies (detecting and) deliberately ignoring any
> versioned entry points created by setup.py. I'd suggest going for a
> rule that if a project defines two entry points with the same
> definitions, where one ends in the pattern -?\d(\.\d)? (in other
> words, -X, -X.Y, X or X.Y) but they are otherwise the same, then
> ignore the versioned one. If we don't do this, we'll likely end up
> with abominations like virtualenv-2.7-3.4.exe.

We need to be careful with that - scripts may end with a version
number unrelated to the Python version. If we can get agreement on
what we want the *default* idiom to look like (more on that below),
then we can start considering the special cases where that idiom
breaks down (like those where the base script name already ends with a
number, perhaps by inserting a leading "-py" into the suffix in those
cases).

> I'm not 100% sure whether you're saying that the installer should
> mandate a versioning policy, or if the user can specify. The
> altinstall case suggests we need a user override option.

I'm biased because there are actually two places where this matters.
It's not just the user scripts directory (everywhere), but also the
system binary directories on Linux distros.

(Note: I haven't run the ideas below by Slavek or Toshio yet, so take
the finer details with a grain of salt - but the general principles
should be right. I don't believe Slavek is making it to PyCon, but
Toshio is, so we should be able to thrash out some more specifics
there)

>From a Linux distro perspective, these versioned scripts should
arguably have different shebang lines - ignoring the parts that would
remain the same, "pip" would correspond to "python", "pip3" to
"python3" and "pip3.4" to "python3.4". Otherwise you
could change the "python" symlink to point somewhere else, and then
have to figure out why "pip" appeared to succeed, but you couldn't
import things you just installed (because they were installed into the
wrong version).

For system wide installs, *which* ones to install should also be
possible to largely automate by following a convention based on the
final name element in sys.executable (and that approach would also fix
the problem of potential conflicts in the user scripts directory).

There's also the fact that for universal (or otherwise cross-version)
wheels, which scripts get installed should be governed by the version
of Python used for the *installation*, rather than the one used to do
the build.

If we put this under the control of the installer, then there is some
hope of eventually bringing order to the chaos and getting some kind
of consistency in naming schemes and alignment with the shebang lines.
If we leave it up to the individual projects? Not a chance (especially
since we *can't* get it right if the versioned scripts are embedded in
wheel files at build time).

> There's also the question of if I do "pip install virtualenv
> --use-versioned X,X.Y" and then "pip install -U virtualenv" will the
> installer remember my versioning choices for the upgrade? Doing so
> needs some complex logic. Not doing so means that "pip install -U pip"
> could change the executables ensurepip had chosen to create.

Yeah, I think driving it based on sys.executable is a better idea,
with a fallback of "-<interpretername>" as the suffix if the final
component of sys.executable doesn't follow one of the recognised
patterns (python, pythonX, pythonX.Y). The reason for using
sys.executable is because that is the thing that can tell us the
interpreter's role *with respect to the larger system*.

Under that model, on a current Fedora system, we would see the
following behaviour when run under the following interpreters:

python -> pip, pip2, pip2.7
python2 -> pip2, pip2.7
python2.7 -> pip2.7
python3 -> pip3, pip3.3
python3.3 -> pip3.3
pypy -> pip-pypy

That avoids collisions not only in /usr/bin (or /usr/local/bin if we
change the install directory for scripts on POSIX), but also in
~/.local/bin (and the per-user directory on Windows).

On Windows system wide installs, since the Scripts directory path
already includes the Python version, I'd just install all 3 in all
cases. Ditto for virtual environments (even when using an alternate
interpreter implementation).

That's essentially using ensurepip's behaviour as precedent for how we
*want* Python scripts to behave.

You may ask "why would you ever do that?", and from my perspective,
the answer is because Fedora is migrating over to using "pip" in the
RPM build process for Python packages, rather than calling setup.py
directly. This means we can more easily adapt to changes in upstream
conventions and automatically take advantage of the upcoming metadata
2.0 enhancements and related installation database changes. However,
it also means we're relying on being able to get pip to play nice with
Fedora's dual stack python2/python3 setup, even as we start migrating
more and more components over to Python 3

The one "user" (i.e. "specfile author") controlled behaviour we would
likely want is the ability to decide whether Python 2 or Python 3 was
considered the default for a given module - when pure applications
that just happen to be written in Python (rather than Python libraries
or Python specific tools) cut over to Python 3, they'll likely just
switch entirely, rather than going the dual stack Python 2/3 approach.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia