On Fri, May 24, 2013 at 10:23 PM, R. David Murray wrote:

Gentoo has a (fairly complex) driver script that is symlinked to all of these bin scripts. The system then has the concept of the "current python", which can be set to python2 or python3. The default bin then calls the current default interpreter. There are also xxx2 and xxx3 versions of each bin script, which call the 'current' version of python2 or python3, respectively.

I'm one of the Gentoo devs, on the python team. I haven't actually written any code for this, but I can show a little of what's going on. I think most of the code is actually in https://bitbucket.org/mgorny/python-exec. We then install three scripts: lrwxrwxrwx 1 root root 11 May 20 14:06 /usr/bin/sphinx-build -> python-exec -rwxr-xr-x 1 root root 311 May 20 14:06 /usr/bin/sphinx-build-python2.7 -rwxr-xr-x 1 root root 311 May 20 14:06 /usr/bin/sphinx-build-python3.2 sphinx-build-python2.7 looks like this: #!/usr/bin/python2.7 # EASY-INSTALL-ENTRY-SCRIPT: 'Sphinx==1.1.3','console_scripts','sphinx-build' __requires__ = 'Sphinx==1.1.3' import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.exit( load_entry_point('Sphinx==1.1.3', 'console_scripts', 'sphinx-build')() ) We now use a python2.7 suffix rather than just a 2.7 suffix because we will install separate wrappers for e.g. pypy1.9 (and we are also prepared to support jython or other implementations at some point). If you have any more questions, I'll try to answer them; or, join #gentoo-python on Freenode, there are generally people hanging out there who know much more about our setup than I do. Cheers, Dirkjan

On Fri, 24 May 2013 15:56:29 -0400 Barry Warsaw wrote:

Here's something that seems to come up from time to time in Debian.

Take a Python application like tox, nose, or pyflakes. Their executables work with both Python 2 and 3, but require a #! line to choose which interpreter to invoke.

When we add Python 3 support in Debian for such a script, all the library installations are handled just fine, but we have conflicts about what to name the thing that lands in /usr/bin. Do we have /usr/bin/pyflakes and /usr/bin/pyflakes3? Do we call the latter py3flakes (as has been convention with some other scripts, but which breaks locate(1))? Do we simply remove the /usr/bin scripts and encourage people to use something like `$python -m nose`?

How about always running the version specific targets, e.g. nosetests-2.7? Regards Antoine.

On Sat, 2013-05-25 at 17:57 +1000, Nick Coghlan wrote:

I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable.

I'm curious if folks have other concrete examples of global bindir executables other than nosetests and pydoc that need to be disambiguated by Python version. I'd hate to see it become standard practice to append "3" to scripts generated by packages which happen to use Python 3, as it will just sort of perpetuate its otherness. - C

On Sat, May 25, 2013 at 05:57:28PM +1000, Nick Coghlan wrote:

On Sat, May 25, 2013 at 5:56 AM, Barry Warsaw wrote:

...

Have any other *nix distros addressed this, and if so, how do you solve it?

I believe Fedora follows the lead set by our own makefile and just appends a "3" to the script name when there is also a Python 2 equivalent (thus ``pydoc3`` and ``pyvenv``). (I don't have any other system provided Python 3 scripts on this machine, though)

Fedora is a bit of a mess... we try to work with upstream's intent when upstream has realized this problem exists and have a single standard when upstream does not. The full guidelines are here: http://fedoraproject.org/wiki/Packaging:Python#Naming Here's the summary: * If the scripts don't care whether they're running on py2 or py3, just use the base name and choose python2 as the interpreter for now (since we can't currently get rid of python2 on an end user system, that is the choice that brings in less dependencies). ex: /usr/bin/pygmentize * If the script does two different things depending on python2 or python3 being the interpreter (note: this includes both bilingual scripts and scripts which have been modified by 2to3/exist in two separate versions) then we have to look at what upstream is doing: - If upstream already deals with it (ex: pydoc3, easy_install-3.1) then we use upstream's name. We don't love this from an inter-package consistently standpoint as there are other packages which append a version for their own usage (is /usr/bin/foo-3.4 for python-3.4 or the 3.4 version of the foo package?) (And we sometimes have to do this locally if we need to have multiple versions of a package with the multiple versions having scripts... ) We decided to use upstream's name if they account for this issue because it will match with upstream's documentation and nothing else seemed as important in this instance. - If upstream doesn't deal with it, then we use a "python3-" prefix. This matches with our package naming so it seemed to make sense. (But Barry's point about locate and tab completion and such would be a reason to revisit this... Perhaps standardizing on /usr/bin/foo2-python3 [pathological case of having both package version and interpreter version in the name.] - (tangent from a different portion of this thread: we've found that this is a larger problem than we would hope. There are some obvious ones like - ipython (implements a python interpreter so python2 vs python3 is understandably important ad different). - nosetests (the python source being operated on is run through the python interpreter so the version has to match). - easy_install (needs to install python modules to the correct interpreter's site-packages. It decides the correct interpreter according to which interpreter invoked it.) But recently we found a new class of problems: frameworks which are bilinugual. For instance, if you have a web framework which has a /usr/bin/django-admin script that can be used to quickstart a project, run a python shell and automatically load your code, load your ORM db schema and operate on it to make modifications to the db then that script has to know whether your code is compatible with python2 or python3.

...

It would be nice if we could have some cross-platform recommendations so things work the same wherever you go. To that end, if we can reach some consensus, I'd be willing to put together an informational PEP and some scripts that might be of general use.

It seems to me the existing recommendation to use ``#!/usr/bin/env python`` instead of referencing a particular binary already covers the general case. The challenge for the distros is that we want a solution that *ignores* user level virtual environments.

I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable.

I'd rather not have a bare 3 for the issues notes above. Something like py3 would be better. There's still room for confusion when distributions have to push multiple versions of a package with scripts that fall into this category. Should the format be: /usr/bin/foo2-py3 (My preference as it places the version next to the thing that it's a version of.) or /usr/bin/foo-py3-2 (Confusing as the 2 is bare. Something like /usr/bin/foo-py3-v2 is slightly better but still not as nice as the previous IMHO) -Toshio

On Tue, 28 May 2013 11:35:00 -0400, Barry Warsaw wrote:

On May 24, 2013, at 04:23 PM, R. David Murray wrote:

...

Gentoo has a (fairly complex) driver script that is symlinked to all of these bin scripts. The system then has the concept of the "current python", which can be set to python2 or python3. The default bin then calls the current default interpreter. There are also xxx2 and xxx3 versions of each bin script, which call the 'current' version of python2 or python3, respectively.

I'm sure one of the gentoo devs on this list can speak to this more completely...I'm just a user :) But I must say that the system works well from my point of view.

Interesting approach, but it doesn't seem to me to be fundamentally different than the BPOS (big pile o' symlinks).

Over in Debian-land one of the interesting points against a driver script was that folks like to be able to explicitly override the shebang line interpreter, e.g.

$ head /usr/bin/foo #! /usr/bin/python3 -Es $ python3.4 /usr/bin/foo ...

One other person mentioned they like to be able to execfile() - or the Python 3 moral equivalent - the /usr/bin script, which obvious would be harder with a sh or binary driver script.

True. Another big disadvantage is that you can't just look in the file to find out what it is doing, which I *do* find be a significant drawback. I have the same complaint about setuptools entry-point scripts, where I still haven't figured out how to go from what is in the file to the code that actually gets called. --David

On May 25, 2013, at 06:17 AM, Chris McDonough wrote:

I'm curious if folks have other concrete examples of global bindir executables other than nosetests and pydoc that need to be disambiguated by Python version. I'd hate to see it become standard practice to append "3" to scripts generated by packages which happen to use Python 3, as it will just sort of perpetuate its otherness.

tox https://bitbucket.org/hpk42/tox/issue/96/cant-have-a-python-3-setuppy -Barry

On May 27, 2013, at 11:38 AM, Toshio Kuratomi wrote:

Fedora is a bit of a mess... we try to work with upstream's intent when upstream has realized this problem exists and have a single standard when upstream does not. The full guidelines are here:

http://fedoraproject.org/wiki/Packaging:Python#Naming

Thanks. One of the reasons I've brought this up here is so that hopefully we can come up with recommendations for upstreams where this matters. One thing is for sure (IMO, anyway). Utilities that provide version-specific scripts should also provide -m invocation. E.g. there are various places where a package's tests (provided unittest, or other as-built tests) can be invoked. Where those might use nose, we recommend invoking them with `$python -m nose` instead using nosetests-X.Y. This also makes it easier to loop over all the versions of Python available on the system (which might not be known statically).

- If upstream doesn't deal with it, then we use a "python3-" prefix. This matches with our package naming so it seemed to make sense. (But Barry's point about locate and tab completion and such would be a reason to revisit this... Perhaps standardizing on /usr/bin/foo2-python3 [pathological case of having both package version and interpreter version in the name.]

Note that the Gentoo example also takes into account versions that might act differently based on the interpreter's implementation. So a -python3 suffix may not be enough. Maybe now we're getting into PEP 425 compatibility tag territory.

- (tangent from a different portion of this thread: we've found that this is a larger problem than we would hope. There are some obvious ones like - ipython (implements a python interpreter so python2 vs python3 is understandably important ad different). - nosetests (the python source being operated on is run through the python interpreter so the version has to match). - easy_install (needs to install python modules to the correct interpreter's site-packages. It decides the correct interpreter according to which interpreter invoked it.)

But recently we found a new class of problems: frameworks which are bilinugual. For instance, if you have a web framework which has a /usr/bin/django-admin script that can be used to quickstart a project, run a python shell and automatically load your code, load your ORM db schema and operate on it to make modifications to the db then that script has to know whether your code is compatible with python2 or python3.

Yay.

...

I think the simplest thing to do is just append the "3" to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable.

I'd rather not have a bare 3 for the issues notes above. Something like py3 would be better.

Same here. I definitely don't like the current Debian semi-convention (not standardized or consistent) of injecting a '3' in the middle of the name, e.g. py3compile or py3doc. Note that adopting PEP 425 conventions allows for -py3 suffix to mean any Python 3 version, compatible across minor version numbers or implementations. This probably translates into a shebang line of #! /usr/bin/python3 whereas -py33 would mean #! /usr/bin/python3.3 This might be overkill in some cases, but at least it builds on existing standards.

There's still room for confusion when distributions have to push multiple versions of a package with scripts that fall into this category. Should the format be:

/usr/bin/foo2-py3 (My preference as it places the version next to the thing that it's a version of.)

or

/usr/bin/foo-py3-2 (Confusing as the 2 is bare. Something like /usr/bin/foo-py3-v2 is slightly better but still not as nice as the previous IMHO)

Definitely the former, especially if PEP 425 serves at the basis for standardization. -Barry

On May 25, 2013, at 03:12 AM, Chris McDonough wrote:

You probably already know this, but I'll mention it anyway. This probably matters a lot for nose and pyflakes, but I'd say that for tox it should not, it basically just scripts execution of shell commands. I'd think maybe in cases like tox (and others that are compatible with both Python 2 and 3) the hashbang should just be set to "#!/usr/bin/python" unconditionally.

Unfortunately, not entirely so: https://bitbucket.org/hpk42/tox/issue/96/cant-have-a-python-3-setuppy

Maybe we could also think about modifying pyflakes so that it can validate both 2 and 3 code (choosing one or the other based on a header line in the validated files and defaulting to the version of Python being run). This is kind of the right thing anyway.

Agreed. Auto-detection may need to be accompanied by a command line option to override in some cases. But I agree, that in general, it would be very nice if the script itself were actually bilingual. (But then, see my previous comment about cross-interpreter dependencies.)

Nose is a bit of a special case. I personally never run nosetests directly, I always use setup.py nosetests, which makes it not matter.

Which is morally equivalent to `$python -m nose`.

In general, I'd like to think that scripts that get installed to global bindirs will execute utilities that are useful independent of the version of Python being used to execute them.

Agreed. I'm trying to tease out some conventions we can recommend for when this can't be the case for whatever reason. -Barry

On Tue, 28 May 2013 13:27:18 -0400 Barry Warsaw wrote:

On May 25, 2013, at 09:53 AM, Antoine Pitrou wrote:

...

How about always running the version specific targets, e.g. nosetests-2.7?

We have nosetests-2.7 and nosetests3 in /usr/bin, but we generally recommend folks not use these, especially for things like (build time) package tests. It's harder to iterate over when the installed versions are unknown statically, e.g. if you wanted to run all the tests over all available versions of Python.

It sounds like you want a dedicated script or utility for this ("run all the tests over all available versions of Python") rather than hack it every time you package a Python library. Your use case also doesn't seem to impact end-users.

This is why I would really like to see all scripts provide a -m equivalent for command line invocation. This might be a little awkward for < Python 2.7 (where IIRC -m doesn't work with packages).

Do you still support Python < 2.7? Regards Antoine.

On Tue, May 28, 2013 at 2:04 PM, Barry Warsaw wrote:

On May 28, 2013, at 01:57 PM, Daniel Holth wrote:

...

Wheel has no mechanism for renaming scripts (or any file) based on the Python version used to install. Instead you would have to build python-version-specific packages for each desired script name.

Note that I'm not trying to borrow any implementation details from wheels, just the file naming conventions (compatibility tags) described in PEP 425. It would still be up to upstream package or distro tools to fiddle the installed file names.

I'm just saying that I prefer a setup.py without too many Python-version-specific differences, since it would look pretty silly to install a wheel of nose generated on Python 3.2 on Python 3.3 and have the wrong version suffix on the scripts. I like the plainly named scripts without version suffixes.

On Tue, 28 May 2013 12:17:49 -0400, Tres Seaver wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 05/28/2013 11:41 AM, R. David Murray wrote:

...

I have the same complaint about setuptools entry-point scripts, where I still haven't figured out how to go from what is in the file to the code that actually gets called.

Hmm, just dump the 'entry_points.txt' file in the named distribution's EGG-INFO directory? E.g.:

$ cat bin/pip #!/path/to/virtualenv/bin/pythonX.Y # EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.3.1','console_scripts','pip' __requires__ = 'pip==1.3.1' import sys from pkg_resources import load_entry_point

if __name__ == '__main__': sys.exit( load_entry_point('pip==1.3.1', 'console_scripts', 'pip')() )

$ cat lib/pythonX.Y/site-packages/pip-1.3.1-pyX.Y.egg/EGG-INFO/entry_points.txt [console_scripts] pip = pip:main pip-X.Y = pip:main

I'm afraid I'm still not enlightened. I'm sure I would understand this if I had ever set up an entry point, since I would have had to read the docs on how to do it. But I never have. So, my point is that the information on what python code is actually being called ought to be in the stub script file, as a comment if nothing else, for discoverability reasons. I'm not bothered enough to work up a patch, though :) --David

On Wed, May 29, 2013 at 5:23 AM, Toshio Kuratomi wrote:

On Tue, May 28, 2013 at 01:22:01PM -0400, Barry Warsaw wrote:

...

On May 27, 2013, at 11:38 AM, Toshio Kuratomi wrote:

...

- If upstream doesn't deal with it, then we use a "python3-" prefix. This matches with our package naming so it seemed to make sense. (But Barry's point about locate and tab completion and such would be a reason to revisit this... Perhaps standardizing on /usr/bin/foo2-python3 [pathological case of having both package version and interpreter version in the name.]

Note that the Gentoo example also takes into account versions that might act differently based on the interpreter's implementation. So a -python3 suffix may not be enough. Maybe now we're getting into PEP 425 compatibility tag territory.

<nod> This is an interesting, unmapped area in Fedora at the moment... I was hoping to talk to Nick and the Fedora python maintainer at our next Fedora conference.

I've been looking at how Fedora's ruby guidelines are implemented wrt alternate interpreters and wondering if we could do something similar for python:

https://fedoraproject.org/wiki/Packaging:Ruby#Different_Interpreters_Compati...

I'm not sure yet how much of that I'd (or Nick and the python maintainer [bkabrda, the current python maintainer is the one who wrote the rubypick script]) would want to use in python -- replacing /usr/bin/python with a script that chooses between CPython and pypy based on user preference gave me an instinctual feeling of dread the first time I looked at it but it seems to be working well for the ruby folks.

My current feeling is that I wouldn't use this same system for interpreters which are not mostly compatible (for instance, python2 vs python3). but I also haven't devoted much actual time to thinking about whether that might have some advantages.

PEP 432 is also related, as it includes the "pysystem" proposal [1] (an alternate Python CLI that will default to -Es behaviour, but is otherwise similar to the standard "python" interpreter). The rest of the discussion though makes me think we may actually need a *nix equivalent of PEP 397 (which describes the "py" launcher we created to get around the limitations of Windows file associations). Between that and the interpreter identification mechanism defined for the PEP 425 compatibility tags it should be possible to come up with an upstream solution for 3.4 that the distros can backport to work with earlier versions (similar to the way users can download the Windows launcher directly from https://bitbucket.org/pypa/pylauncher/downloads even though we only started shipping it upstream as part of the Python 3.3 installer) Cheers, Nick. [1] http://www.python.org/dev/peps/pep-0432/#a-system-python-executable -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On May 28, 2013, at 08:02 PM, Antoine Pitrou wrote:

On Tue, 28 May 2013 13:27:18 -0400 Barry Warsaw wrote:

...

On May 25, 2013, at 09:53 AM, Antoine Pitrou wrote:

...

How about always running the version specific targets, e.g. nosetests-2.7?

We have nosetests-2.7 and nosetests3 in /usr/bin, but we generally recommend folks not use these, especially for things like (build time) package tests. It's harder to iterate over when the installed versions are unknown statically, e.g. if you wanted to run all the tests over all available versions of Python.

It sounds like you want a dedicated script or utility for this ("run all the tests over all available versions of Python") rather than hack it every time you package a Python library.

There is some support for this in some of the Debian helpers, e.g. pybuild, but that's not in widespread use. One problem is that there's no definitive way to know how to run a package's test suite (and won't be even in after PEP 426). I tried to generate some momentum around trying to standardize this, but it didn't get anywhere. Still, that's just one small aspect of the problem.

Your use case also doesn't seem to impact end-users.

Depends on who the end-users are. It definitely impacts developers.

...

This is why I would really like to see all scripts provide a -m equivalent for command line invocation. This might be a little awkward for < Python 2.7 (where IIRC -m doesn't work with packages).

Do you still support Python < 2.7?

Debian still does support 2.6, but hopefully not for long! -Barry

On May 29, 2013, at 01:01 PM, Nick Coghlan wrote:

PEP 432 is also related, as it includes the "pysystem" proposal [1] (an alternate Python CLI that will default to -Es behaviour, but is otherwise similar to the standard "python" interpreter).

I *knew* this was being specified somewhere, but I couldn't find it in either the tracker or PEP summary. As an aside Nick, what do you think about splitting the pysystem proposal out of PEP 432? I think they could certainly live as independent PEPs albeit perhaps the pysystem one dependent on 432.

The rest of the discussion though makes me think we may actually need a *nix equivalent of PEP 397 (which describes the "py" launcher we created to get around the limitations of Windows file associations).

Perhaps!

Between that and the interpreter identification mechanism defined for the PEP 425 compatibility tags it should be possible to come up with an upstream solution for 3.4 that the distros can backport to work with earlier versions (similar to the way users can download the Windows launcher directly from https://bitbucket.org/pypa/pylauncher/downloads even though we only started shipping it upstream as part of the Python 3.3 installer)

We're getting pretty close to a real idea here. :) -Barry