One open question with pip's handling of script generation for wheels is what to do with versioned entry points (virtualenv-2.7.exe etc).
At the moment, pip explicitly disables distlib's facility to generate "versioned" script wrappers for entry points, and generates exactly what the project metadata specifies[1]. The problem is that some projects use sys.version in setup.py to explicitly specify versioned commands. This does not work for wheels, where the build and install environments are different.
There's an open issue for virtualenv, which has hit this problem, and I suspect there are a number of other projects which will hit it if & when they start publishing wheels (I think py.test and nose have the same problem, for a start).
The first question is, who should be responsible for deciding whether to have versioned commands anyway? Is that a project decision, or should it be left to the user doing the install? Should the user be allowed to override a project default?
If it's a user decision, pip probably needs an additional command line argument to specify whether the user wants versioned commands. We may also want pip to ignore any existing entry point specifications that look like they are versioned. (And of course someone needs to tell the projects that currently specify versioned entry points that they should stop :-))
If it's a project decision, we need a means for the project to record that data in metadata that pip can read. For Metadata 2.0, that means something in the console script metadata extension [2]. For the short term, we may need some sort of convention or addition for entry_points.txt. But I'm not sure who would be affected by something like that (beyond the obvious answer of setuptools and pip...)
Does anyone have any thoughts?
Paul
[1] There's a hack in there to generate versioned entry points for pip and easy_install. [2] What's the process for agreeing on extension schemas? Do they need their own PEPs, or what? I was skimming the discussion at the point when these things got pushed out to extensions, so I missed the detail.
On 28 March 2014 14:56, Paul Moore p.f.moore@gmail.com wrote:
[2] What's the process for agreeing on extension schemas? Do they need their own PEPs, or what? I was skimming the discussion at the point when these things got pushed out to extensions, so I missed the detail.
I just found PEP 459. So I guess the question is should the "commands" extension include some sort of "versioned" flag in the wrap_console/wrap_gui data?
Paul
I would like to see just a very simple $ interpolation language. On Mar 28, 2014 11:03 AM, "Paul Moore" p.f.moore@gmail.com wrote:
On 28 March 2014 14:56, Paul Moore p.f.moore@gmail.com wrote:
[2] What's the process for agreeing on extension schemas? Do they need their own PEPs, or what? I was skimming the discussion at the point when these things got pushed out to extensions, so I missed the detail.
I just found PEP 459. So I guess the question is should the "commands" extension include some sort of "versioned" flag in the wrap_console/wrap_gui data?
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 28 March 2014 15:32, Daniel Holth dholth@gmail.com wrote:
I would like to see just a very simple $ interpolation language.
So you think the project should decide, not the user? Or should the user be allowed to override? Paul
On Mar 28, 2014, at 11:51 AM, Paul Moore p.f.moore@gmail.com wrote:
On 28 March 2014 15:32, Daniel Holth dholth@gmail.com wrote:
I would like to see just a very simple $ interpolation language.
So you think the project should decide, not the user? Or should the user be allowed to override? Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
For what it’s worth, if the person invoking the pip command picks that would remove a need for a hack we have in place for ensurepip.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Definitely the project. On Mar 28, 2014 11:51 AM, "Paul Moore" p.f.moore@gmail.com wrote:
On 28 March 2014 15:32, Daniel Holth dholth@gmail.com wrote:
I would like to see just a very simple $ interpolation language.
So you think the project should decide, not the user? Or should the user be allowed to override? Paul
Does anyone have any thoughts?
Note that there is one situation where scripts for multiple Python versions reside in the same directory: per-user site-packages (PEP 370). If you install a package which has scripts and you don't write versioned scripts, a second installation of that package (with a different Python version) will cause the first script to be overwritten. Much of the time this won't matter, but there might be scenarios where it does.
It may also give a problem when uninstalling, or when verifying the installation (as the hash of an overwritten script may not match what's expected, as per the original installation).
To avoid overwriting, one would need to write versioned scripts *only* :-)
Regards,
Vinay Sajip
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/28/2014 01:18 PM, Vinay Sajip wrote:
Does anyone have any thoughts?
Note that there is one situation where scripts for multiple Python versions reside in the same directory: per-user site-packages (PEP 370). If you install a package which has scripts and you don't write versioned scripts, a second installation of that package (with a different Python version) will cause the first script to be overwritten. Much of the time this won't matter, but there might be scenarios where it does.
It may also give a problem when uninstalling, or when verifying the installation (as the hash of an overwritten script may not match what's expected, as per the original installation).
To avoid overwriting, one would need to write versioned scripts *only* :-)
Heh, sounds like a call for Underdog^W:
$ python setup.py altinstall
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
On 29 Mar 2014 06:20, "Vinay Sajip" vinay_sajip@yahoo.co.uk wrote:
Does anyone have any thoughts?
Note that there is one situation where scripts for multiple Python
versions reside in the same directory: per-user site-packages (PEP 370). If you install a package which has scripts and you don't write versioned scripts, a second installation of that package (with a different Python version) will cause the first script to be overwritten. Much of the time this won't matter, but there might be scenarios where it does.
It may also give a problem when uninstalling, or when verifying the
installation (as the hash of an overwritten script may not match what's expected, as per the original installation).
To avoid overwriting, one would need to write versioned scripts *only* :-)
Speaking of which, it's worth describing exactly what we had to do for ensurepip. That has 3 configurations:
POSIX system default: pip3, pip3.4 Windows & venv default: pip, pip3, pip3.4 Alt install: pip3.4
There's currently an issue where "make altinstall && make install" is slightly broken, as it won't add the missing "pip3" command.
So consider me in the school that suggests this be a standard installer feature that can be applied to any entry point script, rather than something that varies by project.
Cheers, Nick.
Regards,
Vinay Sajip _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 28 March 2014 21:06, Nick Coghlan ncoghlan@gmail.com wrote:
So consider me in the school that suggests this be a standard installer feature that can be applied to any entry point script, rather than something that varies by project.
Doing that also implies (detecting and) deliberately ignoring any versioned entry points created by setup.py. I'd suggest going for a rule that if a project defines two entry points with the same definitions, where one ends in the pattern -?\d(.\d)? (in other words, -X, -X.Y, X or X.Y) but they are otherwise the same, then ignore the versioned one. If we don't do this, we'll likely end up with abominations like virtualenv-2.7-3.4.exe.
I'm not 100% sure whether you're saying that the installer should mandate a versioning policy, or if the user can specify. The altinstall case suggests we need a user override option.
There's also the question of if I do "pip install virtualenv --use-versioned X,X.Y" and then "pip install -U virtualenv" will the installer remember my versioning choices for the upgrade? Doing so needs some complex logic. Not doing so means that "pip install -U pip" could change the executables ensurepip had chosen to create.
Paul
On 29 March 2014 07:23, Paul Moore p.f.moore@gmail.com wrote:
On 28 March 2014 21:06, Nick Coghlan ncoghlan@gmail.com wrote:
So consider me in the school that suggests this be a standard installer feature that can be applied to any entry point script, rather than something that varies by project.
Doing that also implies (detecting and) deliberately ignoring any versioned entry points created by setup.py. I'd suggest going for a rule that if a project defines two entry points with the same definitions, where one ends in the pattern -?\d(.\d)? (in other words, -X, -X.Y, X or X.Y) but they are otherwise the same, then ignore the versioned one. If we don't do this, we'll likely end up with abominations like virtualenv-2.7-3.4.exe.
We need to be careful with that - scripts may end with a version number unrelated to the Python version. If we can get agreement on what we want the *default* idiom to look like (more on that below), then we can start considering the special cases where that idiom breaks down (like those where the base script name already ends with a number, perhaps by inserting a leading "-py" into the suffix in those cases).
I'm not 100% sure whether you're saying that the installer should mandate a versioning policy, or if the user can specify. The altinstall case suggests we need a user override option.
I'm biased because there are actually two places where this matters. It's not just the user scripts directory (everywhere), but also the system binary directories on Linux distros.
(Note: I haven't run the ideas below by Slavek or Toshio yet, so take the finer details with a grain of salt - but the general principles should be right. I don't believe Slavek is making it to PyCon, but Toshio is, so we should be able to thrash out some more specifics there)
From a Linux distro perspective, these versioned scripts should
arguably have different shebang lines - ignoring the parts that would remain the same, "pip" would correspond to "python", "pip3" to "python3" and "pip3.4" to "python3.4". Otherwise you could change the "python" symlink to point somewhere else, and then have to figure out why "pip" appeared to succeed, but you couldn't import things you just installed (because they were installed into the wrong version).
For system wide installs, *which* ones to install should also be possible to largely automate by following a convention based on the final name element in sys.executable (and that approach would also fix the problem of potential conflicts in the user scripts directory).
There's also the fact that for universal (or otherwise cross-version) wheels, which scripts get installed should be governed by the version of Python used for the *installation*, rather than the one used to do the build.
If we put this under the control of the installer, then there is some hope of eventually bringing order to the chaos and getting some kind of consistency in naming schemes and alignment with the shebang lines. If we leave it up to the individual projects? Not a chance (especially since we *can't* get it right if the versioned scripts are embedded in wheel files at build time).
There's also the question of if I do "pip install virtualenv --use-versioned X,X.Y" and then "pip install -U virtualenv" will the installer remember my versioning choices for the upgrade? Doing so needs some complex logic. Not doing so means that "pip install -U pip" could change the executables ensurepip had chosen to create.
Yeah, I think driving it based on sys.executable is a better idea, with a fallback of "-<interpretername>" as the suffix if the final component of sys.executable doesn't follow one of the recognised patterns (python, pythonX, pythonX.Y). The reason for using sys.executable is because that is the thing that can tell us the interpreter's role *with respect to the larger system*.
Under that model, on a current Fedora system, we would see the following behaviour when run under the following interpreters:
python -> pip, pip2, pip2.7 python2 -> pip2, pip2.7 python2.7 -> pip2.7 python3 -> pip3, pip3.3 python3.3 -> pip3.3 pypy -> pip-pypy
That avoids collisions not only in /usr/bin (or /usr/local/bin if we change the install directory for scripts on POSIX), but also in ~/.local/bin (and the per-user directory on Windows).
On Windows system wide installs, since the Scripts directory path already includes the Python version, I'd just install all 3 in all cases. Ditto for virtual environments (even when using an alternate interpreter implementation).
That's essentially using ensurepip's behaviour as precedent for how we *want* Python scripts to behave.
You may ask "why would you ever do that?", and from my perspective, the answer is because Fedora is migrating over to using "pip" in the RPM build process for Python packages, rather than calling setup.py directly. This means we can more easily adapt to changes in upstream conventions and automatically take advantage of the upcoming metadata 2.0 enhancements and related installation database changes. However, it also means we're relying on being able to get pip to play nice with Fedora's dual stack python2/python3 setup, even as we start migrating more and more components over to Python 3
The one "user" (i.e. "specfile author") controlled behaviour we would likely want is the ability to decide whether Python 2 or Python 3 was considered the default for a given module - when pure applications that just happen to be written in Python (rather than Python libraries or Python specific tools) cut over to Python 3, they'll likely just switch entirely, rather than going the dual stack Python 2/3 approach.
Cheers, Nick.
On Mar 28, 2014, at 6:51 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 29 March 2014 07:23, Paul Moore p.f.moore@gmail.com wrote:
On 28 March 2014 21:06, Nick Coghlan ncoghlan@gmail.com wrote:
So consider me in the school that suggests this be a standard installer feature that can be applied to any entry point script, rather than something that varies by project.
Doing that also implies (detecting and) deliberately ignoring any versioned entry points created by setup.py. I'd suggest going for a rule that if a project defines two entry points with the same definitions, where one ends in the pattern -?\d(.\d)? (in other words, -X, -X.Y, X or X.Y) but they are otherwise the same, then ignore the versioned one. If we don't do this, we'll likely end up with abominations like virtualenv-2.7-3.4.exe.
We need to be careful with that - scripts may end with a version number unrelated to the Python version. If we can get agreement on what we want the *default* idiom to look like (more on that below), then we can start considering the special cases where that idiom breaks down (like those where the base script name already ends with a number, perhaps by inserting a leading "-py" into the suffix in those cases).
I'm not 100% sure whether you're saying that the installer should mandate a versioning policy, or if the user can specify. The altinstall case suggests we need a user override option.
I'm biased because there are actually two places where this matters. It's not just the user scripts directory (everywhere), but also the system binary directories on Linux distros.
(Note: I haven't run the ideas below by Slavek or Toshio yet, so take the finer details with a grain of salt - but the general principles should be right. I don't believe Slavek is making it to PyCon, but Toshio is, so we should be able to thrash out some more specifics there)
From a Linux distro perspective, these versioned scripts should arguably have different shebang lines - ignoring the parts that would remain the same, "pip" would correspond to "python", "pip3" to "python3" and "pip3.4" to "python3.4". Otherwise you could change the "python" symlink to point somewhere else, and then have to figure out why "pip" appeared to succeed, but you couldn't import things you just installed (because they were installed into the wrong version).
For system wide installs, *which* ones to install should also be possible to largely automate by following a convention based on the final name element in sys.executable (and that approach would also fix the problem of potential conflicts in the user scripts directory).
There's also the fact that for universal (or otherwise cross-version) wheels, which scripts get installed should be governed by the version of Python used for the *installation*, rather than the one used to do the build.
If we put this under the control of the installer, then there is some hope of eventually bringing order to the chaos and getting some kind of consistency in naming schemes and alignment with the shebang lines. If we leave it up to the individual projects? Not a chance (especially since we *can't* get it right if the versioned scripts are embedded in wheel files at build time).
There's also the question of if I do "pip install virtualenv --use-versioned X,X.Y" and then "pip install -U virtualenv" will the installer remember my versioning choices for the upgrade? Doing so needs some complex logic. Not doing so means that "pip install -U pip" could change the executables ensurepip had chosen to create.
Yeah, I think driving it based on sys.executable is a better idea, with a fallback of "-<interpretername>" as the suffix if the final component of sys.executable doesn't follow one of the recognised patterns (python, pythonX, pythonX.Y). The reason for using sys.executable is because that is the thing that can tell us the interpreter's role *with respect to the larger system*.
Under that model, on a current Fedora system, we would see the following behaviour when run under the following interpreters:
python -> pip, pip2, pip2.7 python2 -> pip2, pip2.7 python2.7 -> pip2.7 python3 -> pip3, pip3.3 python3.3 -> pip3.3 pypy -> pip-pypy
That avoids collisions not only in /usr/bin (or /usr/local/bin if we change the install directory for scripts on POSIX), but also in ~/.local/bin (and the per-user directory on Windows).
On Windows system wide installs, since the Scripts directory path already includes the Python version, I'd just install all 3 in all cases. Ditto for virtual environments (even when using an alternate interpreter implementation).
That's essentially using ensurepip's behaviour as precedent for how we *want* Python scripts to behave.
You may ask "why would you ever do that?", and from my perspective, the answer is because Fedora is migrating over to using "pip" in the RPM build process for Python packages, rather than calling setup.py directly. This means we can more easily adapt to changes in upstream conventions and automatically take advantage of the upcoming metadata 2.0 enhancements and related installation database changes. However, it also means we're relying on being able to get pip to play nice with Fedora's dual stack python2/python3 setup, even as we start migrating more and more components over to Python 3
The one "user" (i.e. "specfile author") controlled behaviour we would likely want is the ability to decide whether Python 2 or Python 3 was considered the default for a given module - when pure applications that just happen to be written in Python (rather than Python libraries or Python specific tools) cut over to Python 3, they'll likely just switch entirely, rather than going the dual stack Python 2/3 approach.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
This all sounds reasonable to me. Ultimately the end user should be in control of what gets put on their machine. They are the only one who know which scheme makes sense.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-
Daniel Holth -
Donald Stufft -
Nick Coghlan -
Paul Moore -
Tres Seaver -
Vinay Sajip