Here's something that seems to come up from time to time in Debian. Take a Python application like tox, nose, or pyflakes. Their executables work with both Python 2 and 3, but require a #! line to choose which interpreter to invoke. When we add Python 3 support in Debian for such a script, all the library installations are handled just fine, but we have conflicts about what to name the thing that lands in /usr/bin. Do we have /usr/bin/pyflakes and /usr/bin/pyflakes3? Do we call the latter py3flakes (as has been convention with some other scripts, but which breaks locate(1))? Do we simply remove the /usr/bin scripts and encourage people to use something like `$python -m nose`? One interesting recent suggestion is to create a shell driver script and make that thing accept --py3 and --py2 flags, which would then select the appropriate interpreter and invoke the actual Python driver. There are some technical details to iron out with that, but there's some appeal to it. Have any other *nix distros addressed this, and if so, how do you solve it? It would be nice if we could have some cross-platform recommendations so things work the same wherever you go. To that end, if we can reach some consensus, I'd be willing to put together an informational PEP and some scripts that might be of general use. -Barry
On Fri, 24 May 2013 15:56:29 -0400, Barry Warsaw <barry@python.org> wrote:
Have any other *nix distros addressed this, and if so, how do you solve it? It would be nice if we could have some cross-platform recommendations so things work the same wherever you go. To that end, if we can reach some consensus, I'd be willing to put together an informational PEP and some scripts that might be of general use.
Gentoo has a (fairly complex) driver script that is symlinked to all of these bin scripts. The system then has the concept of the "current python", which can be set to python2 or python3. The default bin then calls the current default interpreter. There are also xxx2 and xxx3 versions of each bin script, which call the 'current' version of python2 or python3, respectively. I'm sure one of the gentoo devs on this list can speak to this more completely...I'm just a user :) But I must say that the system works well from my point of view. --David
On Fri, May 24, 2013 at 10:23 PM, R. David Murray <rdmurray@bitdance.com> wrote:
Gentoo has a (fairly complex) driver script that is symlinked to all of these bin scripts. The system then has the concept of the "current python", which can be set to python2 or python3. The default bin then calls the current default interpreter. There are also xxx2 and xxx3 versions of each bin script, which call the 'current' version of python2 or python3, respectively.
I'm one of the Gentoo devs, on the python team. I haven't actually written any code for this, but I can show a little of what's going on. I think most of the code is actually in https://bitbucket.org/mgorny/python-exec. We then install three scripts: lrwxrwxrwx 1 root root 11 May 20 14:06 /usr/bin/sphinx-build -> python-exec -rwxr-xr-x 1 root root 311 May 20 14:06 /usr/bin/sphinx-build-python2.7 -rwxr-xr-x 1 root root 311 May 20 14:06 /usr/bin/sphinx-build-python3.2 sphinx-build-python2.7 looks like this: #!/usr/bin/python2.7 # EASY-INSTALL-ENTRY-SCRIPT: 'Sphinx==1.1.3','console_scripts','sphinx-build' __requires__ = 'Sphinx==1.1.3' import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.exit( load_entry_point('Sphinx==1.1.3', 'console_scripts', 'sphinx-build')() ) We now use a python2.7 suffix rather than just a 2.7 suffix because we will install separate wrappers for e.g. pypy1.9 (and we are also prepared to support jython or other implementations at some point). If you have any more questions, I'll try to answer them; or, join #gentoo-python on Freenode, there are generally people hanging out there who know much more about our setup than I do. Cheers, Dirkjan
On May 24, 2013, at 04:23 PM, R. David Murray wrote:
Gentoo has a (fairly complex) driver script that is symlinked to all of these bin scripts. The system then has the concept of the "current python", which can be set to python2 or python3. The default bin then calls the current default interpreter. There are also xxx2 and xxx3 versions of each bin script, which call the 'current' version of python2 or python3, respectively.
I'm sure one of the gentoo devs on this list can speak to this more completely...I'm just a user :) But I must say that the system works well from my point of view.
Interesting approach, but it doesn't seem to me to be fundamentally different than the BPOS (big pile o' symlinks). Over in Debian-land one of the interesting points against a driver script was that folks like to be able to explicitly override the shebang line interpreter, e.g. $ head /usr/bin/foo #! /usr/bin/python3 -Es $ python3.4 /usr/bin/foo ... One other person mentioned they like to be able to execfile() - or the Python 3 moral equivalent - the /usr/bin script, which obvious would be harder with a sh or binary driver script. -Barry
On Tue, 28 May 2013 11:35:00 -0400, Barry Warsaw <barry@python.org> wrote:
On May 24, 2013, at 04:23 PM, R. David Murray wrote:
Gentoo has a (fairly complex) driver script that is symlinked to all of these bin scripts. The system then has the concept of the "current python", which can be set to python2 or python3. The default bin then calls the current default interpreter. There are also xxx2 and xxx3 versions of each bin script, which call the 'current' version of python2 or python3, respectively.
I'm sure one of the gentoo devs on this list can speak to this more completely...I'm just a user :) But I must say that the system works well from my point of view.
Interesting approach, but it doesn't seem to me to be fundamentally different than the BPOS (big pile o' symlinks).
Over in Debian-land one of the interesting points against a driver script was that folks like to be able to explicitly override the shebang line interpreter, e.g.
$ head /usr/bin/foo #! /usr/bin/python3 -Es $ python3.4 /usr/bin/foo ...
One other person mentioned they like to be able to execfile() - or the Python 3 moral equivalent - the /usr/bin script, which obvious would be harder with a sh or binary driver script.
True. Another big disadvantage is that you can't just look in the file to find out what it is doing, which I *do* find be a significant drawback. I have the same complaint about setuptools entry-point scripts, where I still haven't figured out how to go from what is in the file to the code that actually gets called. --David
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/28/2013 11:41 AM, R. David Murray wrote:
I have the same complaint about setuptools entry-point scripts, where I still haven't figured out how to go from what is in the file to the code that actually gets called.
Hmm, just dump the 'entry_points.txt' file in the named distribution's EGG-INFO directory? E.g.: $ cat bin/pip #!/path/to/virtualenv/bin/pythonX.Y # EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.3.1','console_scripts','pip' __requires__ = 'pip==1.3.1' import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.exit( load_entry_point('pip==1.3.1', 'console_scripts', 'pip')() ) $ cat lib/pythonX.Y/site-packages/pip-1.3.1-pyX.Y.egg/EGG-INFO/entry_points.txt [console_scripts] pip = pip:main pip-X.Y = pip:main Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGk2K0ACgkQ+gerLs4ltQ6WaACZAbdz7k3sdM21DNx0mzcecY93 hvYAoJTwA2l3OvSoYStzGmsJ+N16JDwM =YHcy -----END PGP SIGNATURE-----
On Tue, 28 May 2013 12:17:49 -0400, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 05/28/2013 11:41 AM, R. David Murray wrote:
I have the same complaint about setuptools entry-point scripts, where I still haven't figured out how to go from what is in the file to the code that actually gets called.
Hmm, just dump the 'entry_points.txt' file in the named distribution's EGG-INFO directory? E.g.:
$ cat bin/pip #!/path/to/virtualenv/bin/pythonX.Y # EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.3.1','console_scripts','pip' __requires__ = 'pip==1.3.1' import sys from pkg_resources import load_entry_point
if __name__ == '__main__': sys.exit( load_entry_point('pip==1.3.1', 'console_scripts', 'pip')() )
$ cat lib/pythonX.Y/site-packages/pip-1.3.1-pyX.Y.egg/EGG-INFO/entry_points.txt [console_scripts] pip = pip:main pip-X.Y = pip:main
I'm afraid I'm still not enlightened. I'm sure I would understand this if I had ever set up an entry point, since I would have had to read the docs on how to do it. But I never have. So, my point is that the information on what python code is actually being called ought to be in the stub script file, as a comment if nothing else, for discoverability reasons. I'm not bothered enough to work up a patch, though :) --David
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/28/2013 05:52 PM, R. David Murray wrote:
On Tue, 28 May 2013 12:17:49 -0400, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 05/28/2013 11:41 AM, R. David Murray wrote:
I have the same complaint about setuptools entry-point scripts, where I still haven't figured out how to go from what is in the file to the code that actually gets called.
Hmm, just dump the 'entry_points.txt' file in the named distribution's EGG-INFO directory? E.g.:
$ cat bin/pip #!/path/to/virtualenv/bin/pythonX.Y # EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.3.1','console_scripts','pip' __requires__ = 'pip==1.3.1' import sys from pkg_resources import load_entry_point
if __name__ == '__main__': sys.exit( load_entry_point('pip==1.3.1', 'console_scripts', 'pip')() )
$ cat lib/pythonX.Y/site-packages/pip-1.3.1-pyX.Y.egg/EGG-INFO/entry_points.txt
[console_scripts]
pip = pip:main pip-X.Y = pip:main
I'm afraid I'm still not enlightened.
I'm sure I would understand this if I had ever set up an entry point, since I would have had to read the docs on how to do it. But I never have.
So, my point is that the information on what python code is actually being called ought to be in the stub script file, as a comment if nothing else, for discoverability reasons.
I'm not bothered enough to work up a patch, though :)
It is there already: # EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.3.1','console_scripts','pip' Which says, load the entry point named 'pip' from the 'console_scripts' entry point group in the 'pip 1.3.1' distribution. The 'entry_points.txt' metadata file specifies that that entry point is a function named 'main' inside the 'pip' package itself. Ters. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEUEARECAAYFAlGlZesACgkQ+gerLs4ltQ50xACeJUBMjAvMBaOm63Viigz2bvkP S5gAl2w4WAxgasXie10DMtHJOyRRFvA= =34KH -----END PGP SIGNATURE-----
On Tue, 28 May 2013 22:20:33 -0400, Tres Seaver <tseaver@palladion.com> wrote:
So, my point is that the information on what python code is actually being called ought to be in the stub script file, as a comment if nothing else, for discoverability reasons.
I'm not bothered enough to work up a patch, though :)
It is there already:
# EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.3.1','console_scripts','pip'
Which says, load the entry point named 'pip' from the 'console_scripts' entry point group in the 'pip 1.3.1' distribution.
The 'entry_points.txt' metadata file specifies that that entry point is a function named 'main' inside the 'pip' package itself.
Ah, but you had to *decode* that for me, using your non-local expert's knowledge. I assume 'main' is defined in or imported into pip's __init__? Now, if the comment had said: # Call pip.main (per the specification in the pip entry of the # console_scripts section of pip-1.3.1-egg-info/entrypoints.txt). then I would have known everything I needed to know without either consulting the *implementor's* documentation for setuptools or an expert such as yourself. Of that, as a *user*, the first two words are the only thing I'm interested in, but the other information could be handy in debugging certain specialized and unlikely issues, such as when someone has manually changed the entrypoints.txt file. Note that the comment still requires you to know python import semantics...but if you don't know that much you wouldn't get far looking at the source code anyway. The dir/filename lets you 'find' the entrypoints.txt file even if you don't know where on your system that file is installed. --David
On Fri, 2013-05-24 at 15:56 -0400, Barry Warsaw wrote:
Here's something that seems to come up from time to time in Debian.
Take a Python application like tox, nose, or pyflakes. Their executables work with both Python 2 and 3, but require a #! line to choose which interpreter to invoke.
You probably already know this, but I'll mention it anyway. This probably matters a lot for nose and pyflakes, but I'd say that for tox it should not, it basically just scripts execution of shell commands. I'd think maybe in cases like tox (and others that are compatible with both Python 2 and 3) the hashbang should just be set to "#!/usr/bin/python" unconditionally. Maybe we could also think about modifying pyflakes so that it can validate both 2 and 3 code (choosing one or the other based on a header line in the validated files and defaulting to the version of Python being run). This is kind of the right thing anyway. Nose is a bit of a special case. I personally never run nosetests directly, I always use setup.py nosetests, which makes it not matter. In general, I'd like to think that scripts that get installed to global bindirs will execute utilities that are useful independent of the version of Python being used to execute them. - C
On May 25, 2013, at 03:12 AM, Chris McDonough wrote:
You probably already know this, but I'll mention it anyway. This probably matters a lot for nose and pyflakes, but I'd say that for tox it should not, it basically just scripts execution of shell commands. I'd think maybe in cases like tox (and others that are compatible with both Python 2 and 3) the hashbang should just be set to "#!/usr/bin/python" unconditionally.
Unfortunately, not entirely so: https://bitbucket.org/hpk42/tox/issue/96/cant-have-a-python-3-setuppy
Maybe we could also think about modifying pyflakes so that it can validate both 2 and 3 code (choosing one or the other based on a header line in the validated files and defaulting to the version of Python being run). This is kind of the right thing anyway.
Agreed. Auto-detection may need to be accompanied by a command line option to override in some cases. But I agree, that in general, it would be very nice if the script itself were actually bilingual. (But then, see my previous comment about cross-interpreter dependencies.)
Nose is a bit of a special case. I personally never run nosetests directly, I always use setup.py nosetests, which makes it not matter.
Which is morally equivalent to `$python -m nose`.
In general, I'd like to think that scripts that get installed to global bindirs will execute utilities that are useful independent of the version of Python being used to execute them.
Agreed. I'm trying to tease out some conventions we can recommend for when this can't be the case for whatever reason. -Barry
On Tue, May 28, 2013 at 1:30 PM, Barry Warsaw <barry@python.org> wrote:
On May 25, 2013, at 03:12 AM, Chris McDonough wrote:
You probably already know this, but I'll mention it anyway. This probably matters a lot for nose and pyflakes, but I'd say that for tox it should not, it basically just scripts execution of shell commands. I'd think maybe in cases like tox (and others that are compatible with both Python 2 and 3) the hashbang should just be set to "#!/usr/bin/python" unconditionally.
Unfortunately, not entirely so:
https://bitbucket.org/hpk42/tox/issue/96/cant-have-a-python-3-setuppy
Maybe we could also think about modifying pyflakes so that it can validate both 2 and 3 code (choosing one or the other based on a header line in the validated files and defaulting to the version of Python being run). This is kind of the right thing anyway.
Agreed. Auto-detection may need to be accompanied by a command line option to override in some cases. But I agree, that in general, it would be very nice if the script itself were actually bilingual. (But then, see my previous comment about cross-interpreter dependencies.)
Nose is a bit of a special case. I personally never run nosetests directly, I always use setup.py nosetests, which makes it not matter.
Which is morally equivalent to `$python -m nose`.
In general, I'd like to think that scripts that get installed to global bindirs will execute utilities that are useful independent of the version of Python being used to execute them.
Agreed. I'm trying to tease out some conventions we can recommend for when this can't be the case for whatever reason.
-Barry
Wheel has no mechanism for renaming scripts (or any file) based on the Python version used to install. Instead you would have to build python-version-specific packages for each desired script name.
On May 28, 2013, at 01:57 PM, Daniel Holth wrote:
Wheel has no mechanism for renaming scripts (or any file) based on the Python version used to install. Instead you would have to build python-version-specific packages for each desired script name.
Note that I'm not trying to borrow any implementation details from wheels, just the file naming conventions (compatibility tags) described in PEP 425. It would still be up to upstream package or distro tools to fiddle the installed file names. -Barry
On Tue, May 28, 2013 at 2:04 PM, Barry Warsaw <barry@python.org> wrote:
On May 28, 2013, at 01:57 PM, Daniel Holth wrote:
Wheel has no mechanism for renaming scripts (or any file) based on the Python version used to install. Instead you would have to build python-version-specific packages for each desired script name.
Note that I'm not trying to borrow any implementation details from wheels, just the file naming conventions (compatibility tags) described in PEP 425. It would still be up to upstream package or distro tools to fiddle the installed file names.
I'm just saying that I prefer a setup.py without too many Python-version-specific differences, since it would look pretty silly to install a wheel of nose generated on Python 3.2 on Python 3.3 and have the wrong version suffix on the scripts. I like the plainly named scripts without version suffixes.
On Fri, 24 May 2013 15:56:29 -0400 Barry Warsaw <barry@python.org> wrote:
Here's something that seems to come up from time to time in Debian.
Take a Python application like tox, nose, or pyflakes. Their executables work with both Python 2 and 3, but require a #! line to choose which interpreter to invoke.
When we add Python 3 support in Debian for such a script, all the library installations are handled just fine, but we have conflicts about what to name the thing that lands in /usr/bin. Do we have /usr/bin/pyflakes and /usr/bin/pyflakes3? Do we call the latter py3flakes (as has been convention with some other scripts, but which breaks locate(1))? Do we simply remove the /usr/bin scripts and encourage people to use something like `$python -m nose`?
How about always running the version specific targets, e.g. nosetests-2.7? Regards Antoine.
On May 25, 2013, at 09:53 AM, Antoine Pitrou wrote:
How about always running the version specific targets, e.g. nosetests-2.7?
We have nosetests-2.7 and nosetests3 in /usr/bin, but we generally recommend folks not use these, especially for things like (build time) package tests. It's harder to iterate over when the installed versions are unknown statically, e.g. if you wanted to run all the tests over all available versions of Python. For those, we recommend people use `$python -m nose` since the available versions of Python can be queried from the system. This is why I would really like to see all scripts provide a -m equivalent for command line invocation. This might be a little awkward for < Python 2.7 (where IIRC -m doesn't work with packages). -Barry
On Tue, 28 May 2013 13:27:18 -0400 Barry Warsaw <barry@python.org> wrote:
On May 25, 2013, at 09:53 AM, Antoine Pitrou wrote:
How about always running the version specific targets, e.g. nosetests-2.7?
We have nosetests-2.7 and nosetests3 in /usr/bin, but we generally recommend folks not use these, especially for things like (build time) package tests. It's harder to iterate over when the installed versions are unknown statically, e.g. if you wanted to run all the tests over all available versions of Python.
It sounds like you want a dedicated script or utility for this ("run all the tests over all available versions of Python") rather than hack it every time you package a Python library. Your use case also doesn't seem to impact end-users.
This is why I would really like to see all scripts provide a -m equivalent for command line invocation. This might be a little awkward for < Python 2.7 (where IIRC -m doesn't work with packages).
Do you still support Python < 2.7? Regards Antoine.
On May 28, 2013, at 08:02 PM, Antoine Pitrou wrote:
On Tue, 28 May 2013 13:27:18 -0400 Barry Warsaw <barry@python.org> wrote:
On May 25, 2013, at 09:53 AM, Antoine Pitrou wrote:
How about always running the version specific targets, e.g. nosetests-2.7?
We have nosetests-2.7 and nosetests3 in /usr/bin, but we generally recommend folks not use these, especially for things like (build time) package tests. It's harder to iterate over when the installed versions are unknown statically, e.g. if you wanted to run all the tests over all available versions of Python.
It sounds like you want a dedicated script or utility for this ("run all the tests over all available versions of Python") rather than hack it every time you package a Python library.
There is some support for this in some of the Debian helpers, e.g. pybuild, but that's not in widespread use. One problem is that there's no definitive way to know how to run a package's test suite (and won't be even in after PEP 426). I tried to generate some momentum around trying to standardize this, but it didn't get anywhere. Still, that's just one small aspect of the problem.
Your use case also doesn't seem to impact end-users.
Depends on who the end-users are. It definitely impacts developers.
This is why I would really like to see all scripts provide a -m equivalent for command line invocation. This might be a little awkward for < Python 2.7 (where IIRC -m doesn't work with packages).
Do you still support Python < 2.7?
Debian still does support 2.6, but hopefully not for long! -Barry
On Sat, May 25, 2013 at 5:56 AM, Barry Warsaw <barry@python.org> wrote:
Have any other *nix distros addressed this, and if so, how do you solve it?
I believe Fedora follows the lead set by our own makefile and just appends a "3" to the script name when there is also a Python 2 equivalent (thus ``pydoc3`` and ``pyvenv``). (I don't have any other system provided Python 3 scripts on this machine, though)
It would be nice if we could have some cross-platform recommendations so things work the same wherever you go. To that end, if we can reach some consensus, I'd be willing to put together an informational PEP and some scripts that might be of general use.
It seems to me the existing recommendation to use ``#!/usr/bin/env python`` instead of referencing a particular binary already covers the general case. The challenge for the distros is that we want a solution that *ignores* user level virtual environments. I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 2013-05-25 at 17:57 +1000, Nick Coghlan wrote:
I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable.
I'm curious if folks have other concrete examples of global bindir executables other than nosetests and pydoc that need to be disambiguated by Python version. I'd hate to see it become standard practice to append "3" to scripts generated by packages which happen to use Python 3, as it will just sort of perpetuate its otherness. - C
On Sat, May 25, 2013 at 8:17 PM, Chris McDonough <chrism@plope.com> wrote:
On Sat, 2013-05-25 at 17:57 +1000, Nick Coghlan wrote:
I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable.
I'm curious if folks have other concrete examples of global bindir executables other than nosetests and pydoc that need to be disambiguated by Python version.
Single source Python 2/3 packages don't have the problem. They can either run with the system Python explicitly, or use "/usr/bin/env python" in order to respect virtual environments. The issue only exists for projects where Python 2 and Python 3 are separate code bases with distinct scripts to be installed on the target system. In my opinion, is just one more reason why single source is a vastly superior alternative to treating the Python 2 and Python 3 versions as separate applications.
I'd hate to see it become standard practice to append "3" to scripts generated by packages which happen to use Python 3, as it will just sort of perpetuate its otherness.
Fedora only does it for stuff that has to straddle the two with a parallel install as a system binary. If something is exclusive to Py3 (like "pyvenv") it doesn't get the suffix. It's certainly not an elegant solution, but Python is far from the only runtime platform afflicted by similar issues when it comes to balancing the interests of distro developers wanting to build a single integrated system against those of application developers wanting to use more recent versions of their dependencies. Longer term, we (Red Hat) want to better support application stacks that are more independent of the system versions, while still being fully under the control of the system package manager (to ease security updates). The mechanism underlying that is Software Collections, which is better described here: http://developerblog.redhat.com/2013/01/28/software-collections-on-red-hat-e... (The software collection system also exists in Fedora, as that's where it was developed, but the main benefit there is to allow application developers to use something *older* than what Fedora provides in the system packages if there are backwards compatibility issues with an update) I have no idea if Debian are contemplating anything similar, but the current situation where application developers have to choose between letting distro vendors control the upgrade cycle of key dependencies and divorcing themselves entirely from the benefits offered by the system package manager is unsustainable in the long term. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On May 25, 2013, at 06:17 AM, Chris McDonough wrote:
I'm curious if folks have other concrete examples of global bindir executables other than nosetests and pydoc that need to be disambiguated by Python version. I'd hate to see it become standard practice to append "3" to scripts generated by packages which happen to use Python 3, as it will just sort of perpetuate its otherness.
tox https://bitbucket.org/hpk42/tox/issue/96/cant-have-a-python-3-setuppy -Barry
On Sat, May 25, 2013 at 05:57:28PM +1000, Nick Coghlan wrote:
On Sat, May 25, 2013 at 5:56 AM, Barry Warsaw <barry@python.org> wrote:
Have any other *nix distros addressed this, and if so, how do you solve it?
I believe Fedora follows the lead set by our own makefile and just appends a "3" to the script name when there is also a Python 2 equivalent (thus ``pydoc3`` and ``pyvenv``). (I don't have any other system provided Python 3 scripts on this machine, though)
Fedora is a bit of a mess... we try to work with upstream's intent when upstream has realized this problem exists and have a single standard when upstream does not. The full guidelines are here: http://fedoraproject.org/wiki/Packaging:Python#Naming Here's the summary: * If the scripts don't care whether they're running on py2 or py3, just use the base name and choose python2 as the interpreter for now (since we can't currently get rid of python2 on an end user system, that is the choice that brings in less dependencies). ex: /usr/bin/pygmentize * If the script does two different things depending on python2 or python3 being the interpreter (note: this includes both bilingual scripts and scripts which have been modified by 2to3/exist in two separate versions) then we have to look at what upstream is doing: - If upstream already deals with it (ex: pydoc3, easy_install-3.1) then we use upstream's name. We don't love this from an inter-package consistently standpoint as there are other packages which append a version for their own usage (is /usr/bin/foo-3.4 for python-3.4 or the 3.4 version of the foo package?) (And we sometimes have to do this locally if we need to have multiple versions of a package with the multiple versions having scripts... ) We decided to use upstream's name if they account for this issue because it will match with upstream's documentation and nothing else seemed as important in this instance. - If upstream doesn't deal with it, then we use a "python3-" prefix. This matches with our package naming so it seemed to make sense. (But Barry's point about locate and tab completion and such would be a reason to revisit this... Perhaps standardizing on /usr/bin/foo2-python3 [pathological case of having both package version and interpreter version in the name.] - (tangent from a different portion of this thread: we've found that this is a larger problem than we would hope. There are some obvious ones like - ipython (implements a python interpreter so python2 vs python3 is understandably important ad different). - nosetests (the python source being operated on is run through the python interpreter so the version has to match). - easy_install (needs to install python modules to the correct interpreter's site-packages. It decides the correct interpreter according to which interpreter invoked it.) But recently we found a new class of problems: frameworks which are bilinugual. For instance, if you have a web framework which has a /usr/bin/django-admin script that can be used to quickstart a project, run a python shell and automatically load your code, load your ORM db schema and operate on it to make modifications to the db then that script has to know whether your code is compatible with python2 or python3.
It would be nice if we could have some cross-platform recommendations so things work the same wherever you go. To that end, if we can reach some consensus, I'd be willing to put together an informational PEP and some scripts that might be of general use.
It seems to me the existing recommendation to use ``#!/usr/bin/env python`` instead of referencing a particular binary already covers the general case. The challenge for the distros is that we want a solution that *ignores* user level virtual environments.
I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable.
I'd rather not have a bare 3 for the issues notes above. Something like py3 would be better. There's still room for confusion when distributions have to push multiple versions of a package with scripts that fall into this category. Should the format be: /usr/bin/foo2-py3 (My preference as it places the version next to the thing that it's a version of.) or /usr/bin/foo-py3-2 (Confusing as the 2 is bare. Something like /usr/bin/foo-py3-v2 is slightly better but still not as nice as the previous IMHO) -Toshio
On May 27, 2013, at 11:38 AM, Toshio Kuratomi wrote:
Fedora is a bit of a mess... we try to work with upstream's intent when upstream has realized this problem exists and have a single standard when upstream does not. The full guidelines are here:
Thanks. One of the reasons I've brought this up here is so that hopefully we can come up with recommendations for upstreams where this matters. One thing is for sure (IMO, anyway). Utilities that provide version-specific scripts should also provide -m invocation. E.g. there are various places where a package's tests (provided unittest, or other as-built tests) can be invoked. Where those might use nose, we recommend invoking them with `$python -m nose` instead using nosetests-X.Y. This also makes it easier to loop over all the versions of Python available on the system (which might not be known statically).
- If upstream doesn't deal with it, then we use a "python3-" prefix. This matches with our package naming so it seemed to make sense. (But Barry's point about locate and tab completion and such would be a reason to revisit this... Perhaps standardizing on /usr/bin/foo2-python3 [pathological case of having both package version and interpreter version in the name.]
Note that the Gentoo example also takes into account versions that might act differently based on the interpreter's implementation. So a -python3 suffix may not be enough. Maybe now we're getting into PEP 425 compatibility tag territory.
- (tangent from a different portion of this thread: we've found that this is a larger problem than we would hope. There are some obvious ones like - ipython (implements a python interpreter so python2 vs python3 is understandably important ad different). - nosetests (the python source being operated on is run through the python interpreter so the version has to match). - easy_install (needs to install python modules to the correct interpreter's site-packages. It decides the correct interpreter according to which interpreter invoked it.)
But recently we found a new class of problems: frameworks which are bilinugual. For instance, if you have a web framework which has a /usr/bin/django-admin script that can be used to quickstart a project, run a python shell and automatically load your code, load your ORM db schema and operate on it to make modifications to the db then that script has to know whether your code is compatible with python2 or python3.
Yay.
I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable.
I'd rather not have a bare 3 for the issues notes above. Something like py3 would be better.
Same here. I definitely don't like the current Debian semi-convention (not standardized or consistent) of injecting a '3' in the middle of the name, e.g. py3compile or py3doc. Note that adopting PEP 425 conventions allows for -py3 suffix to mean any Python 3 version, compatible across minor version numbers or implementations. This probably translates into a shebang line of #! /usr/bin/python3 whereas -py33 would mean #! /usr/bin/python3.3 This might be overkill in some cases, but at least it builds on existing standards.
There's still room for confusion when distributions have to push multiple versions of a package with scripts that fall into this category. Should the format be:
/usr/bin/foo2-py3 (My preference as it places the version next to the thing that it's a version of.)
or
/usr/bin/foo-py3-2 (Confusing as the 2 is bare. Something like /usr/bin/foo-py3-v2 is slightly better but still not as nice as the previous IMHO)
Definitely the former, especially if PEP 425 serves at the basis for standardization. -Barry
On Tue, May 28, 2013 at 01:22:01PM -0400, Barry Warsaw wrote:
On May 27, 2013, at 11:38 AM, Toshio Kuratomi wrote:
- If upstream doesn't deal with it, then we use a "python3-" prefix. This matches with our package naming so it seemed to make sense. (But Barry's point about locate and tab completion and such would be a reason to revisit this... Perhaps standardizing on /usr/bin/foo2-python3 [pathological case of having both package version and interpreter version in the name.]
Note that the Gentoo example also takes into account versions that might act differently based on the interpreter's implementation. So a -python3 suffix may not be enough. Maybe now we're getting into PEP 425 compatibility tag territory.
<nod> This is an interesting, unmapped area in Fedora at the moment... I was hoping to talk to Nick and the Fedora python maintainer at our next Fedora conference. I've been looking at how Fedora's ruby guidelines are implemented wrt alternate interpreters and wondering if we could do something similar for python: https://fedoraproject.org/wiki/Packaging:Ruby#Different_Interpreters_Compati... I'm not sure yet how much of that I'd (or Nick and the python maintainer [bkabrda, the current python maintainer is the one who wrote the rubypick script]) would want to use in python -- replacing /usr/bin/python with a script that chooses between CPython and pypy based on user preference gave me an instinctual feeling of dread the first time I looked at it but it seems to be working well for the ruby folks. My current feeling is that I wouldn't use this same system for interpreters which are not mostly compatible (for instance, python2 vs python3). but I also haven't devoted much actual time to thinking about whether that might have some advantages. -Toshio
On Wed, May 29, 2013 at 5:23 AM, Toshio Kuratomi <a.badger@gmail.com> wrote:
On Tue, May 28, 2013 at 01:22:01PM -0400, Barry Warsaw wrote:
On May 27, 2013, at 11:38 AM, Toshio Kuratomi wrote:
- If upstream doesn't deal with it, then we use a "python3-" prefix. This matches with our package naming so it seemed to make sense. (But Barry's point about locate and tab completion and such would be a reason to revisit this... Perhaps standardizing on /usr/bin/foo2-python3 [pathological case of having both package version and interpreter version in the name.]
Note that the Gentoo example also takes into account versions that might act differently based on the interpreter's implementation. So a -python3 suffix may not be enough. Maybe now we're getting into PEP 425 compatibility tag territory.
<nod> This is an interesting, unmapped area in Fedora at the moment... I was hoping to talk to Nick and the Fedora python maintainer at our next Fedora conference.
I've been looking at how Fedora's ruby guidelines are implemented wrt alternate interpreters and wondering if we could do something similar for python:
https://fedoraproject.org/wiki/Packaging:Ruby#Different_Interpreters_Compati...
I'm not sure yet how much of that I'd (or Nick and the python maintainer [bkabrda, the current python maintainer is the one who wrote the rubypick script]) would want to use in python -- replacing /usr/bin/python with a script that chooses between CPython and pypy based on user preference gave me an instinctual feeling of dread the first time I looked at it but it seems to be working well for the ruby folks.
My current feeling is that I wouldn't use this same system for interpreters which are not mostly compatible (for instance, python2 vs python3). but I also haven't devoted much actual time to thinking about whether that might have some advantages.
PEP 432 is also related, as it includes the "pysystem" proposal [1] (an alternate Python CLI that will default to -Es behaviour, but is otherwise similar to the standard "python" interpreter). The rest of the discussion though makes me think we may actually need a *nix equivalent of PEP 397 (which describes the "py" launcher we created to get around the limitations of Windows file associations). Between that and the interpreter identification mechanism defined for the PEP 425 compatibility tags it should be possible to come up with an upstream solution for 3.4 that the distros can backport to work with earlier versions (similar to the way users can download the Windows launcher directly from https://bitbucket.org/pypa/pylauncher/downloads even though we only started shipping it upstream as part of the Python 3.3 installer) Cheers, Nick. [1] http://www.python.org/dev/peps/pep-0432/#a-system-python-executable -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On May 29, 2013, at 01:01 PM, Nick Coghlan wrote:
PEP 432 is also related, as it includes the "pysystem" proposal [1] (an alternate Python CLI that will default to -Es behaviour, but is otherwise similar to the standard "python" interpreter).
I *knew* this was being specified somewhere, but I couldn't find it in either the tracker or PEP summary. As an aside Nick, what do you think about splitting the pysystem proposal out of PEP 432? I think they could certainly live as independent PEPs albeit perhaps the pysystem one dependent on 432.
The rest of the discussion though makes me think we may actually need a *nix equivalent of PEP 397 (which describes the "py" launcher we created to get around the limitations of Windows file associations).
Perhaps!
Between that and the interpreter identification mechanism defined for the PEP 425 compatibility tags it should be possible to come up with an upstream solution for 3.4 that the distros can backport to work with earlier versions (similar to the way users can download the Windows launcher directly from https://bitbucket.org/pypa/pylauncher/downloads even though we only started shipping it upstream as part of the Python 3.3 installer)
We're getting pretty close to a real idea here. :) -Barry
On 30 May 2013 04:40, "Barry Warsaw" <barry@python.org> wrote:
On May 29, 2013, at 01:01 PM, Nick Coghlan wrote:
PEP 432 is also related, as it includes the "pysystem" proposal [1] (an alternate Python CLI that will default to -Es behaviour, but is otherwise similar to the standard "python" interpreter).
I *knew* this was being specified somewhere, but I couldn't find it in
either
the tracker or PEP summary. As an aside Nick, what do you think about splitting the pysystem proposal out of PEP 432? I think they could certainly live as independent PEPs albeit perhaps the pysystem one dependent on 432.
Sure. You could probably even implement it without PEP 432, it would just be somewhat painful to replicate the current CLI behaviour. Cheers, Nick.
The rest of the discussion though makes me think we may actually need a *nix equivalent of PEP 397 (which describes the "py" launcher we created to get around the limitations of Windows file associations).
Perhaps!
Between that and the interpreter identification mechanism defined for the PEP 425 compatibility tags it should be possible to come up with an upstream solution for 3.4 that the distros can backport to work with earlier versions (similar to the way users can download the Windows launcher directly from https://bitbucket.org/pypa/pylauncher/downloads even though we only started shipping it upstream as part of the Python 3.3 installer)
We're getting pretty close to a real idea here. :)
-Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On May 28, 2013, at 12:23 PM, Toshio Kuratomi wrote:
Note that the Gentoo example also takes into account versions that might act differently based on the interpreter's implementation. So a -python3 suffix may not be enough. Maybe now we're getting into PEP 425 compatibility tag territory.
<nod> This is an interesting, unmapped area in Fedora at the moment... I was hoping to talk to Nick and the Fedora python maintainer at our next Fedora conference.
I've been looking at how Fedora's ruby guidelines are implemented wrt alternate interpreters and wondering if we could do something similar for python:
https://fedoraproject.org/wiki/Packaging:Ruby#Different_Interpreters_Compati...
Very interesting. It was something like this, albeit replacing _jruby_ or _mri_ with something like --py2 or --py3 that I had in mind. However...
I'm not sure yet how much of that I'd (or Nick and the python maintainer [bkabrda, the current python maintainer is the one who wrote the rubypick script]) would want to use in python -- replacing /usr/bin/python with a script that chooses between CPython and pypy based on user preference gave me an instinctual feeling of dread the first time I looked at it but it seems to be working well for the ruby folks.
... it kind of gives me the heebie-jeebies too. I think *most* scripts wouldn't need this kind of variability though. For example, lsb_release only needs to run with one version of Python, so: % head -1 /usr/bin/lsb_release #! /usr/bin/python3 -Es is just fine. I wouldn't want to replace /usr/bin/python with a selectable interpreter (see also PEP 394), but if we had something like /usr/bin/multipy which acted like rubypick for the few, very limited examples where it's needed, then it might be useful to do so. I would very definitely want to get consensus on the mechanism and api between the various Linux distros here so it works the same on F/RH, D/U, Gentoo and any others.
My current feeling is that I wouldn't use this same system for interpreters which are not mostly compatible (for instance, python2 vs python3). but I also haven't devoted much actual time to thinking about whether that might have some advantages.
Seems like for Python, that would be the most important use case, but maybe I have blinders on for the issue at hand. -Barry
On May 25, 2013, at 05:57 PM, Nick Coghlan wrote:
It seems to me the existing recommendation to use ``#!/usr/bin/env python`` instead of referencing a particular binary already covers the general case. The challenge for the distros is that we want a solution that *ignores* user level virtual environments.
Right. My general recommendation is that upstream's (development version) scripts use #! /usr/bin/env, but that distros and possibly even virtualenv/buildout installs, hardcode the #! to a specific interpreter. We've just had way too many problems when a /usr/bin script uses /usr/bin/env and breaks the world. We also recommend using -Es to isolate the environment as much as possible. -Barry
participants (9)
-
Antoine Pitrou -
Barry Warsaw -
Chris McDonough -
Daniel Holth -
Dirkjan Ochtman -
Nick Coghlan -
R. David Murray -
Toshio Kuratomi -
Tres Seaver