2011-03-01 22:50:34 Kerrick Staley napisał(a):

On Tue, Mar 1, 2011 at 3:26 PM, Eric Smith <eric@trueblade.com> wrote:

...

On 3/1/2011 4:19 PM, Kerrick Staley wrote:

...

Hello, There is a need for the default Python2 install to place a symlink at /usr/bin/python2 that points to /usr/bin/python, or for the documentation to recommend that packagers ensure that python2 is defined. Also, all documentation should be changed to recommend that "#!/usr/bin/env python2" be used as the shebang for Python 2 scripts. This is needed because some distributions (Arch Linux, in particular), point /usr/bin/python to /usr/bin/python3, while others (including Slackware, Debian, and the BSDs, probably more) do not even define the python2 command. This means that a script has no way of achieving cross-platform compatibility. The point at which many distributions begin to alias /usr/bin/python to /usr/bin/python3 is due soon, and for the next couple of years, it would be best to use a python2 or python3 shebang in all scripts, making no assumptions about plain python, which should only be invoked interactively. This email from about 3 years ago seems relevant: : http://mail.python.org/pipermail/python-3000/2008-March/012421.html Again, this issue needs to be addressed by the Python developers themselves so that different *nix distributions will handle it consistently, allowing Python scripts to continue to be cross-platform.

I believe we agreed at the language summit last year (or maybe even the year before) that "python" would always be python2.x, and "python3" would be python3.x.

And by "always" we indeed meant forever. To do otherwise would break scripts even many, many years from now.

I understand, but is it at least possible to officially recommend that python, python2, and python3 all exist, that distributions point python to python2, and that scripts specify which of python2 and python3 they are using? This would create a redundant system that doesn't avoids problems even if distributions do decide to ignore the second point. If not, can someone point me to official documentation that recommends that python always invoke Python2, so that I can take the case up with the Arch developers?

Gentoo most likely will switch /usr/bin/python to Python 3 in this year. Majority of Python-2-only packages have been already prepared to use /usr/bin/python2 or /usr/bin/python2.X. -- Arfrever Frehtes Taifersar Arahesis

On Mar 1, 2011, at 5:06 PM, Guido van Rossum wrote:

On Tue, Mar 1, 2011 at 1:26 PM, Eric Smith <eric@trueblade.com> wrote:

...

On 3/1/2011 4:19 PM, Kerrick Staley wrote:

...

Hello, There is a need for the default Python2 install to place a symlink at /usr/bin/python2 that points to /usr/bin/python, or for the documentation to recommend that packagers ensure that python2 is defined. Also, all documentation should be changed to recommend that "#!/usr/bin/env python2" be used as the shebang for Python 2 scripts. This is needed because some distributions (Arch Linux, in particular), point /usr/bin/python to /usr/bin/python3, while others (including Slackware, Debian, and the BSDs, probably more) do not even define the python2 command. This means that a script has no way of achieving cross-platform compatibility. The point at which many distributions begin to alias /usr/bin/python to /usr/bin/python3 is due soon, and for the next couple of years, it would be best to use a python2 or python3 shebang in all scripts, making no assumptions about plain python, which should only be invoked interactively. This email from about 3 years ago seems relevant: : http://mail.python.org/pipermail/python-3000/2008-March/012421.html Again, this issue needs to be addressed by the Python developers themselves so that different *nix distributions will handle it consistently, allowing Python scripts to continue to be cross-platform.

I believe we agreed at the language summit last year (or maybe even the year before) that "python" would always be python2.x, and "python3" would be python3.x.

And by "always" we indeed meant forever. To do otherwise would break scripts even many, many years from now.

Unfortunately distros are not following these guidelines. As long as we still have the pythonX.Y links I think it's better to have "python2", "python3" and "python" than total anarchy.

If python upstream would make it clear that that *IS* the policy, distros might follow it. Right now, there is no clear guidance, as far as I can tell. If you do not want distros making python be a link to python3, please say so loudly, preferably on a webpage on python.org that users can point the distros to. James

Am 02.03.2011 00:16, schrieb Kerrick Staley:

I think that it's a good idea to not only state that python should be Python 2, but also that python2 should be implemented and that scripts should specify it, to provide redundancy and handle distros that won't or have not yet switched back to the python -> python2 convention. I've . In any event, I do agree that there needs to be a clear standard coming from the Python community. Should I submit a PEP for this?

If you want this to become policy: yes, you will have to specify a PEP for that. Even a PEP might not make it policy (practicality beats purity), but not having a PEP guarantees that it won't become policy. Regards, Martin

On Tue, 01 Mar 2011 16:26:05 -0500, Eric Smith <eric@trueblade.com> wrote:

On 3/1/2011 4:19 PM, Kerrick Staley wrote:

...

Hello, There is a need for the default Python2 install to place a symlink at /usr/bin/python2 that points to /usr/bin/python, or for the documentation to recommend that packagers ensure that python2 is defined. Also, all documentation should be changed to recommend that "#!/usr/bin/env python2" be used as the shebang for Python 2 scripts. This is needed because some distributions (Arch Linux, in particular), point /usr/bin/python to /usr/bin/python3, while others (including Slackware, Debian, and the BSDs, probably more) do not even define the python2 command. This means that a script has no way of achieving cross-platform compatibility. The point at which many distributions begin to alias /usr/bin/python to /usr/bin/python3 is due soon, and for the next couple of years, it would be best to use a python2 or python3 shebang in all scripts, making no assumptions about plain python, which should only be invoked interactively. This email from about 3 years ago seems relevant: : http://mail.python.org/pipermail/python-3000/2008-March/012421.html Again, this issue needs to be addressed by the Python developers themselves so that different *nix distributions will handle it consistently, allowing Python scripts to continue to be cross-platform.

I believe we agreed at the language summit last year (or maybe even the year before) that "python" would always be python2.x, and "python3" would be python3.x.

And by "always" we indeed meant forever. To do otherwise would break scripts even many, many years from now.

It sounds like the distributions aren't going to cooperate with us. Arch has already switched. Gentoo will allow the user to switch /usr/bin/python to point to python3, and I suspect this will become the default at some point. I'm not sad about that, myself. --David

On Tue, Mar 1, 2011 at 3:45 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:

...

...

I believe we agreed at the language summit last year (or maybe even the year before) that "python" would always be python2.x, and "python3" would be python3.x.

And by "always" we indeed meant forever. To do otherwise would break scripts even many, many years from now.

It sounds like the distributions aren't going to cooperate with us. Arch has already switched.  Gentoo will allow the user to switch /usr/bin/python to point to python3, and I suspect this will become the default at some point.

I'm not sad about that, myself.

Neither am I. I personally also disagree with the decision taken at the language summit (and believe that the language summit is no place to make such decisions).

I don't recall which side I was on at the time, but now I agree we should try to encourage distros to use python2 for Python 2.x and python for whatever they like to promote. I think a PEP would help, but in this case I would request that before the PEP gets written (it can be a really short one!) somebody actually go out and get consensus from a number of important distros. Besides Barry, do we have any representatives of distros here? -- --Guido van Rossum (python.org/~guido)

On Wed, Mar 02, 2011 at 01:14:32AM +0100, "Martin v. Löwis" wrote:

...

I think a PEP would help, but in this case I would request that before the PEP gets written (it can be a really short one!) somebody actually go out and get consensus from a number of important distros. Besides Barry, do we have any representatives of distros here?

Matthias Klose represents Debian, Dave Malcolm represents Redhat, and Dirkjan Ochtman represents Gentoo.

I'm here from Fedora. -Toshio

On Wed, Mar 2, 2011 at 10:01, Piotr Ożarowski <piotr@debian.org> wrote:

I co-maintain with Matthias a package that provides /usr/bin/python symlink in Debian and I can confirm that it will always point to Python 2.X. We also do not plan to add /usr/bin/python2 symlink (and I guess only accepted PEP can change that)

Can you please explain why you NACK this proposed change? Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi

On Wed, Mar 2, 2011 at 13:56, Piotr Ożarowski <piotr@debian.org> wrote:

[Sandro Tosi, 2011-03-02]

...

On Wed, Mar 2, 2011 at 10:01, Piotr Ożarowski <piotr@debian.org> wrote:

...

I co-maintain with Matthias a package that provides /usr/bin/python symlink in Debian and I can confirm that it will always point to Python 2.X. We also do not plan to add /usr/bin/python2 symlink (and I guess only accepted PEP can change that)

Can you please explain why you NACK this proposed change?

it encourages people to change /usr/bin/python symlink to point to python3.X which I'm strongly against (how can I tell that upstream author meant python3.X and not python2.X without checking the code?)

with 'people' do you mean 'users'? if so, isn't this risk already present? If you, user, change the python symlink (provided by python-minimal in Debian) to something else than what's shipped, it's still a local change, and will never be supported; but with python2 *Debian is free* to decide if python can be pointed to python3, if the time will come. Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi

On 03/03/11 00:03, Piotr Ożarowski wrote:

[Sandro Tosi, 2011-03-02]

...

On Wed, Mar 2, 2011 at 13:56, Piotr Ożarowski<piotr@debian.org> wrote:

...

[Sandro Tosi, 2011-03-02]

...

On Wed, Mar 2, 2011 at 10:01, Piotr Ożarowski<piotr@debian.org> wrote:

...

I co-maintain with Matthias a package that provides /usr/bin/python symlink in Debian and I can confirm that it will always point to Python 2.X. We also do not plan to add /usr/bin/python2 symlink (and I guess only accepted PEP can change that)

Can you please explain why you NACK this proposed change?

it encourages people to change /usr/bin/python symlink to point to python3.X which I'm strongly against (how can I tell that upstream author meant python3.X and not python2.X without checking the code?)

with 'people' do you mean 'users'? if so, isn't this risk already present?

users already break their systems via "sudo ez_install ..." (note the "sudo" part!), I meant developers (distro and upstream authors). If a programmer develops a script in Python 3 on Arch and later ships his file with /usr/bin/python in shebang, it's very likely that this script will not work on all distributions that didn't (yet?) change the symlink.

...

If you, user, change the python symlink (provided by python-minimal in Debian) to something else than what's shipped, it's still a local change, and will never be supported; but with python2 *Debian is free* to decide if python can be pointed to python3, if the time will come.

... and make other distributions developers' life miserable?

But is that not the whole point of adding the /usr/bin/python2 symlink. That way a developer can explicitly use a /usr/bin/python2 or /usr/bin/python3 shebang and have it portable everywhere. At the moment, Debian seems to be the major hold-up on that actually being a reality being the only major distro I could find that does not provide such a symlink. Note also that even restricting /usr/bin/python to point at a python-2.x binary gives no guarantee on what actual python-2.x version you are getting, so it is not as if guaranteeing portability is not a problem already... Allan

On 03/03/11 00:29, Piotr Ożarowski wrote:

[Allan McRae, 2011-03-02]

...

But is that not the whole point of adding the /usr/bin/python2 symlink. That way a developer can explicitly use a /usr/bin/python2 or /usr/bin/python3 shebang and have it portable everywhere. At the moment, Debian seems to be the major hold-up on that actually being a reality being the only major distro I could find that does not provide such a symlink.

Do you realize how many (still perfectly usable) scripts written in Python 2.x few years ago (and not modified since then) are out there? Do you realize how much work would it require to fix every single one of them to point to /usr/bin/python2 instead? Even if we'd start checking mdate and change it at build time automatically, there still will be way too many false positives... for no clear gain.

Having made the packages using python-2.x code from an entire distribution point at /usr/bin/python2, I have a fair idea of how much work is involved... And that is exactly why changes need made now so that time is available for transition. Providing the /usr/bin/python2 symlink now means that any future code would be able to point to it rather than some unversioned python binary. That way in ?? years when python-3.x is "the" python and python-2.x is obsolete, and it is decided that /usr/bin/python will be python-3.x (which I believe is the only logical outcome), then everyone will be a lot more prepared. Allan

On Wed, 02 Mar 2011 10:13:59 -0500, James Y Knight <foom@fuhm.net> wrote:

On Mar 2, 2011, at 9:54 AM, Allan McRae wrote:

...

That way in ?? years when python-3.x is "the" python and python-2.x is obsolete, and it is decided that /usr/bin/python will be python-3.x (which I believe is the only logical outcome),

But that's not the only logical outcome. A perfectly logical outcome is that /usr/bin/python disappears completely if python2.X isn't installed, and python3 is always called python3. That is the outcome I find sensible. And that is the crux of the disagreement in this thread.

Well, I personally won't use a distribution that makes this choice. For whatever that's worth :) But, even if a distribution *does* make that choice, if it wants to be compatible with code developed on distributions that make the other choice, it should provide a /usr/bin/python2 symlink. Otherwise, it is going to be getting bug reports from users asking why XYZ script doesn't run. In short, I don't see any *downside* to providing a /usr/bin/python2 symlink. -- R. David Murray www.bitdance.com

The point is that there never has to be an agreement about the python command, as long as all distros support python2/python3 and all scripts use it (I think that the distinction should continue to be made if/when python2 becomes uncommon, otherwise we'll hit the same issue with python4). We don't have to force anyone to change the python command itself. That being said, I feel that the python command should only be invoked from an interactive terminal, and in fact it would be good if distros would use the python2/python3 convention in their own packages, so that the sysadmin can point python to wherever he wants, and when he types python into a terminal get the version he wants. As an added plus, should a distro supporting this feature decide to make the Python 2 -> Python 3 switch, it would be an effortless process. Again, however, force this requirement shouldn't be forced on distros. As an aside, this whole thing started when I tried installing ROS<http://www.ros.org/wiki/>, only to find that it made assumptions about /usr/bin/python, which points to python3 on my Arch Linux system. I went in search of documentation for the python2/python3 convention so that I could make the ROS developers aware of it and help them to follow it, only to find out that no such convention actually exists. Since ROS uses a rather complicated python-based installer that makes assumptions about /usr/bin/python not only in the shebangs of many files but also in other places (apparently in connection with the subprocess module), it has proven thus far unworkable unless I change /usr/bin/python back to Python 2, a move that could potentially break many other aspects of my system. I'm sure there are many other users out there that are frustrated by similar issues; supporting a python2/python3 convention on all distros as well as in scripts would solve these issues without creating further problems and without the need for a slow consensus to be reached on what /usr/bin/python should be. Here is a draft PEP (forgive me if it's incorrectly formatted; I've never done this before). I think it's a little long winded given how simple the idea it proposes is, but I thought it would be better to be more specific than necessary rather than less. . PEP: ??? Title: The python Command on Unix-Like Systems Version: ??? Last-Modified: ??? Author: Kerrick Staley <mail at kerrickstaley.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Mar-2011 Post-History: ??? Abstract ========== This PEP provides a convention to ensure that Python scripts can continue to be portable across *nix systems, regardless of the default version of the Python interpreter (i.e. the version invoked by the "python" command). Recommendation ================ * ``*nix`` software distributions should install the command "python2" into the default path whenever a version of the Python 2 interpreter is installed, and the same for "python3" and the Python 3 interpreter. When invoked, "python2" should run some version of the Python 2 interpreter, and "python3" should run some version of the Python 3 interpreter. * All new code that needs to invoke the Python interpreter should specify either "python2" or "python3", according to the version it requires, but not "python". This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context. Rationale =========== This is needed because some distributions alias the "python" command to Python 3, while others alias it to Python 2. Some distributions also do not provide a "python2" command; hence, there is no way for Python 2 code (or any code that invokes the Python 2 interpreter) to reliably run on all systems without modification, because both the "python" and the "python2" commands will fail on some systems. The recommendations in this PEP provide a very simple mechanism to restore cross-platform support, with very little additional work required on the part of distribution maintainers. Notes ======= * Distributions can alias the "python" command to whichever version of the Python interpreter they choose. * It would be wise for distributions to always follow the convention that this PEP recommends, even in code that is not intended to operate on other distributions. This will make it easier if the code ever needs to be ported to another distribution or if the distribution decides to change the version of the Python interpreter that the "python" command invokes. Distributions can test whether they are fully following this convention by switching the "python" interpreter and seeing if anything breaks. * If the above point is adhered to, then the "python" command should always be a link to the interpreter binary (or a link to a link) and not vice versa. That way, if users decide to change where the "python" command points, they can do so without inadvertently deleting the binary. * The first recommendation can be ignored for systems on which the python command itself has traditionally been left undefined and users have always had the responsibility of linking the "python" command to the Python interpreter. * If the Python 2 interpreter becomes uncommon, scripts should nevertheless continue to use the "python3" convention rather that just "python". This will ease transition in the event that yet another major version of Python is released. * If these conventions are adhered to, it will be the case that the "python" command is only executed in an interactive manner. Backwards Compatibility ========================= A potential problem can arise if a script adhering to the "python2"/"python3" convention is executed on a system not supporting these commands. This is mostly a non-issue, since the sysadmin can simply create these symbolic links and avoid further problems. Copyright =========== This document has been placed in the public domain. EOF -Kerrick Staley On Wed, Mar 2, 2011 at 9:13 AM, James Y Knight <foom@fuhm.net> wrote:

On Mar 2, 2011, at 9:54 AM, Allan McRae wrote:

...

That way in ?? years when python-3.x is "the" python and python-2.x is obsolete, and it is decided that /usr/bin/python will be python-3.x (which I believe is the only logical outcome),

But that's not the only logical outcome. A perfectly logical outcome is that /usr/bin/python disappears completely if python2.X isn't installed, and python3 is always called python3. That is the outcome I find sensible. And that is the crux of the disagreement in this thread.

Those who think python3.X should stay /usr/bin/python3 forever do not see any reason to make everyone rewrite their existing python scripts to say "/usr/bin/python2" instead of "/usr/bin/python". So, there's no point in adding a /usr/bin/python2 now. Scripts that want python2 can remain using /usr/bin/python forever, and that will either be installed, or not installed, depending on whether that OS has a copy of python2.X.

Those who think python3 should (eventually someday, or maybe immediately, depending) be named or have an alias of "/usr/bin/python" want to make everyone rewrite their scripts to say /usr/bin/python2 now. For that position, it's unfortunate that python source doesn't install itself with an alias of /usr/bin/python2, and some distros don't install that alias either. So they want to fix that.

James _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/mail%40kerrickstaley.com

Here is a draft PEP (forgive me if it's incorrectly formatted; I've never done this before).

LGTM. Please specify what /usr/bin/python is supposed to be also (rather: the "python" utility). I'd like it ruled out that installations *only* provide python2 and python3 - "python" could be either one, but should be present "normally" (i.e. SHOULD in the RFC 2119 sense). Nitpickingly, I'd add that scripts can, of course, also specify python2.7 (or some such). Actually, scripts can do whatever they want - it's more about what they then can expect to happen. Regards, Martin

/tangent Does this discussion remind anyone else of the bash/dash switch for /usr/bin/sh in Ubuntu? The distro itself coped fine, but 3rd party shell scripts that used bash extensions were a whole different story. (No, I'm not sure what lessons, if any, we can draw from that. It just struck me as an interesting parallel worth mentioning) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Wed, 2011-03-02 at 16:20 +0100, Piotr Ożarowski wrote:

[Allan McRae, 2011-03-02]

...

Having made the packages using python-2.x code from an entire distribution point at /usr/bin/python2, I have a fair idea of how much work is involved...

* is every Arch package that uses Python 2.X already working with /usr/bin/python and why not? ;-) * how many Python packages do Arch have in the first place and why so little? ;-) * how does Arch deal with scripts that are not packaged, what do you say to users who report bugs against your packages because their local scripts do not work anymore? :-|

Is that a silver spoon in your mouth?

On Mar 2, 2011, at 12:14 PM, Barry Warsaw wrote:

I don't have a problem with adding such a symlink, and I think it should be done by Informational PEP, not Standards Track PEP. Since there will be no Python 2.8, our own build system shouldn't ever be changed to add such a link, but we can recommend it for consistency among distros, which would be free to adopt it or not.

Why not? 2.7 is supposed to be in long term maintenance mode. Surely if it's a good idea for everyone else to ship a python2 binary, 2.7.next should also install it when building from source... James

Am 02.03.2011 20:49, schrieb James Y Knight:

On Mar 2, 2011, at 12:14 PM, Barry Warsaw wrote:

...

I don't have a problem with adding such a symlink, and I think it should be done by Informational PEP, not Standards Track PEP. Since there will be no Python 2.8, our own build system shouldn't ever be changed to add such a link, but we can recommend it for consistency among distros, which would be free to adopt it or not.

Why not? 2.7 is supposed to be in long term maintenance mode. Surely if it's a good idea for everyone else to ship a python2 binary, 2.7.next should also install it when building from source...

I agree with Barry that this would be a new feature, and, by default, cannot be added to the 2.7 release which is in maintenance mode. IMO, an accepted PEP could override the policy, though. Regards, Martin

On Mar 2, 2011, at 8:23 AM, Sandro Tosi wrote:

On Wed, Mar 2, 2011 at 13:56, Piotr Ożarowski <piotr@debian.org> wrote:

...

[Sandro Tosi, 2011-03-02]

...

On Wed, Mar 2, 2011 at 10:01, Piotr Ożarowski <piotr@debian.org> wrote:

...

I co-maintain with Matthias a package that provides /usr/bin/python symlink in Debian and I can confirm that it will always point to Python 2.X. We also do not plan to add /usr/bin/python2 symlink (and I guess only accepted PEP can change that)

Can you please explain why you NACK this proposed change?

it encourages people to change /usr/bin/python symlink to point to python3.X which I'm strongly against (how can I tell that upstream author meant python3.X and not python2.X without checking the code?)

with 'people' do you mean 'users'? if so, isn't this risk already present?

If you, user, change the python symlink (provided by python-minimal in Debian) to something else than what's shipped, it's still a local change, and will never be supported; but with python2 *Debian is free* to decide if python can be pointed to python3, if the time will come.

I suspect he's saying it'd be better if the time didn't come (if so, I'd agree). Python3 *is* unfortunately a new and incompatible programming language, it makes sense for it to have it have its own interpreter name. Eventually /usr/bin/python might no longer be installed, but that doesn't mean python3 shouldn't simply be called python3 forever. James

James Y Knight wrote:

I suspect he's saying it'd be better if the time didn't come (if so, I'd agree). Python3 *is* unfortunately a new and incompatible programming language, it makes sense for it to have it have its own interpreter name.

Oh come on, there's like three incompatibilities versus three thousand things that are compatible. A pox on your discontinuous mind! *wink* Seriously, most of the changes are library changes, not language changes. The similarities far outweigh the differences. I don't think there is a generally agreed upon objective boundary between "dialect" and "language", but to my mind the change between 2.x and 3.x falls squarely under "dialect". In the same way, I don't care that William Shakespeare's everyday speech would be nearly incomprehensible to my ears, and mine to his, we both speak (spoke?) English. In my opinion, the biggest change from Python 2 -> 3 is that we actually provide developers tools for migrating scripts rather than leave it for them to deal with the changes themselves. I recently ported a client's application written for 2.3 that used string exceptions everywhere to 2.6. I would have loved a "2.3to2.6" fixer :) I don't consider 2.3 and 2.6 to be different languages, and I suspect neither do you, even though code that runs fine without even a warning under one raises a SyntaxError under the other. Every time we drop or rename a module from the standard library, we break scripts. Such backwards incompatibility is not enough to delineate different languages. Even syntax changes are not necessarily enough.

Eventually /usr/bin/python might no longer be installed, but that doesn't mean python3 shouldn't simply be called python3 forever.

I already call Python 3 "python" in casual conversation. There is *no way* that I will be calling it "python3" in fifteen years time, when Python 2.7 is as dead and forgotten as Python 1.5 is now, just to satisfy some overly strict definition of "different language". -- Steven

Jérôme Radix wrote:

Hello,

Defensive programming will force you to do things like :

import sys if sys.version[0] == '2':

Really? Do you already do this? if sys.version < '2.2': result = apply(func, arguments) else: result = func(*arguments) And if so, have you tested it in Python 1.5 to see what happens? -- Steven

No, I don't do it now. But taking like granted the fact that 2.x python will be dead in 5 years and that /usr/bin/python will point to python3 is, imho, a little too optimistic. Thus, as time passes, python scripts will have to guess if they are running through python3 or python2 because the two will be installed on most systems, with no strict convention on how to run a 2.x python script or a 3.x python script. /usr/bin/python is meant to point to python3 one time or another. The time will never be the same for all distros. And, yes, people are already testing python version in their scripts<http://www.google.com/search?q=python+version+test+site%3Astackoverflow.com&ie=utf-8> . As a matter of fact, they've already had this kind of discussion for Perl : they use require <http://perldoc.perl.org/functions/require.html>. Jérôme. 2011/3/2 Steven D'Aprano <steve@pearwood.info>

Steven D'Aprano wrote:

...

Jérôme Radix wrote:

...

Hello,

Defensive programming will force you to do things like :

import sys if sys.version[0] == '2':

Really? Do you already do this?

if sys.version < '2.2': result = apply(func, arguments) else: result = func(*arguments)

And if so, have you tested it in Python 1.5 to see what happens?

Sorry, that reads harsher than I intended. Please insert a wink and a smiley.

-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jerome.radix%40gmail.com

On Wed, Mar 2, 2011 at 3:00 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:

Am 02.03.2011 23:36, schrieb Jérôme Radix:

...

No, I don't do it now. But taking like granted the fact that 2.x python will be dead in 5 years and that /usr/bin/python will point to python3 is, imho, a little too optimistic.

I don't think Steven said, or assumed, a scope of 5 years - more like a scope of 30 years. In 30 years, Python 2 will surely be dead (as will likely be Python 3).

The defensive programming you promote is likely to fail. Many Python 2 scripts are syntactically invalid when interpreted as Python 3, so a version test won't even be executed.

With separate python2 and python3 executables, people can have scripts depend on the right binary.

In interactive mode, I would like to use /usr/bin/python be the "current" Python binary always (even when Python 4.6 comes along). Python will, interactively, greet me with its version number, and I can adjust. So the idea of /usr/bin/python being reserved for Python 2 strikes me as inconvenient.

+1 on all that. -- --Guido van Rossum (python.org/~guido)

[Guido van Rossum, 2011-03-02]

On Wed, Mar 2, 2011 at 4:56 AM, Piotr Ożarowski <piotr@debian.org> wrote:

...

[Sandro Tosi, 2011-03-02]

...

On Wed, Mar 2, 2011 at 10:01, Piotr Ożarowski <piotr@debian.org> wrote:

...

I co-maintain with Matthias a package that provides /usr/bin/python symlink in Debian and I can confirm that it will always point to Python 2.X. We also do not plan to add /usr/bin/python2 symlink (and I guess only accepted PEP can change that)

Can you please explain why you NACK this proposed change?

it encourages people to change /usr/bin/python symlink to point to python3.X which I'm strongly against (how can I tell that upstream author meant python3.X and not python2.X without checking the code?)

But the same is already true for python2.X vs. python2.Y. Explicit is better than implicit etc. Plus, 5 years from now everybody is going to be annoyed that "python" still refers to some ancient unused version of Python.

I don't really mind adding /usr/bin/python2 symlink just to clean Arch mess, but I do mind changing /usr/bin/python to point to python3 (and I can use the same argument - "Explicit is better than implicit" - if you need Python 3, say so in the shebang, right?). What I'm afraid of is when we'll add /usr/bin/python2, we'll start getting a lot of scripts that will have to be checked manually every time new upstream version is released because we cannot assume what upstream author is using at given point. If /usr/bin/python will be disallowed in shebangs on the other hand (and all scripts will use /usr/bin/python2, /usr/bin/python3, /usr/bin/python4 or /usr/bin/python2.6 etc.) I don't see a problem with letting administrators choose /usr/bin/python (right now not only changing it from python2.X to python3.X will break the system but also changing it from /usr/bin/pytohn2.X to /usr/bin/python2.Y will break it, and believe me, I know what I'm talking about (one of the guys at work did something like this once)) [all IMHO, dunno if other Debian's python-defaults maintainers agree with me] -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

On Mar 03, 2011, at 09:08 AM, Toshio Kuratomi wrote:

Thinking outside of the box, I can think of something that would satisfy your requirements but I don't know how appropriate it is for upstream python to ship with. Stop shipping /usr/bin/python. Ship "python" in an alternate location like $LIBEXECDIR/python2.7/bin (I think this would be /usr/lib/python2.7/bin on Debian and /usr/libexec/python2.7/bin on Fedora which would both be appropriate) then configure which python version is invoked by the user typing "python" by configuring PATH (a shell alias might also work). You could configure this with environment-modules[1]_ if Debian supports using that in packaging.

I wonder if Debian's alternatives system would be appropriate for this? http://wiki.debian.org/DebianAlternatives -Barry

On Mar 03, 2011, at 08:37 PM, Toshio Kuratomi wrote:

No, alternatives is really only useful for a very small class of problems [1]_ and [2]_.

Thanks for the clarification. I was on the fence about making the suggestion in the first place. ;)

For this discussion there's an additional problem which is that alternatives works by creating symlinks. Piotr Ożarowski wants to make /usr/bin/python not exist so that scripts would have to use either /usr/bin/python3 or /usr/bin/python2. If alternatives places a symlink there, it defeats the purpose of avoiding that path in the package itself.

I don't agree that /usr/bin/python should not be installed. The draft PEP language hits the right tone IMHO, and I would favor /usr/bin/python pointing to /usr/bin/python2 on Debian, but primarily used only for the interactive interpreter. Or IOW, I still want users to be able to type 'python' at a shell prompt and get the interpreter. Cheers, -Barry

On Mar 04, 2011, at 12:00 PM, Toshio Kuratomi wrote:

Actually, my post was saying that these two can be decoupled. ie: It's possible to not have /usr/bin/python while still allowing users to type python at a shell prompt and get the interpreter.

This is done by either redefining the PATH to include the directory that the interpreter named "python" is in or by creating an alias for python to the proper interpreter.

I personally would prefer aliasing rather than $PATH manipulation. -Barry

On Tue, Mar 08, 2011 at 08:25:50AM +1000, Nick Coghlan wrote:

On Tue, Mar 8, 2011 at 1:30 AM, Barry Warsaw <barry@python.org> wrote:

...

On Mar 04, 2011, at 12:00 PM, Toshio Kuratomi wrote:

...

Actually, my post was saying that these two can be decoupled.  ie: It's possible to not have /usr/bin/python while still allowing users to type python at a shell prompt and get the interpreter.

This is done by either redefining the PATH to include the directory that the interpreter named "python" is in or by creating an alias for python to the proper interpreter.

I personally would prefer aliasing rather than $PATH manipulation.

Toshio's suggestion wouldn't work anyway - the "/usr/bin/env python" idiom will pick up a "python" alias no matter where it lives on $PATH.

I thought I pointed out that env wouldn't work with PATH but I guess I just thought that silently in my head. Pointing that out was going to live in the same paragraph as saying that it does work with an alias:: $ sudo mv /usr/bin/python /usr/bin/python.bak $ alias python='/usr/bin/python2.7' $ python --version Python 2.7 $ cat test.py #! /bin/env python print 'hi' $ ./test.py /bin/env: python: No such file or directory $ mv /usr/bin/python.bak /usr/bin/python $ ./test.py hi -Toshio

On Mar 3, 2011, at 3:55 AM, Piotr Ożarowski wrote:

I don't really mind adding /usr/bin/python2 symlink just to clean Arch mess

Is there any chance you would add the symlink in the next Debian stable point release? If both Debian and Python upstream added the python2 symlink in the next stable update rather than waiting for the next major release (which for Python itself is of course "never"), that could reduce the problem of systems not having the symlink installed quite significantly. James

-On [20110302 01:17], "Martin v. Löwis" (martin@v.loewis.de) wrote:

Matthias Klose represents Debian, Dave Malcolm represents Redhat, and Dirkjan Ochtman represents Gentoo.

With FreeBSD's ports if you install a Python port it will install a pythonX.Y in /usr/local/bin, depending on what is specified with the make variable PYTHON_DEFAULT_VERSION "python" will point to that version. So it can refer to either 2.x or 3.x. NetBSD's pkgsrc does not have 3.x in the tree yet. But if no "python" exists yet, then pkg_alternatives will link it to the version of choice. Not sure what either OpenBSD or DragonFly BSD are doing, but it will be along these lines as well. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | GPG: 2EAC625B Ain't gonna spend the rest of my Life, quietly fading away...

On Thu, 2011-03-03 at 14:17 -0500, David Malcolm wrote:

On Wed, 2011-03-02 at 01:14 +0100, "Martin v. Löwis" wrote:

There are a number of other rpm packages with names matching "*py*", which use the system build of Python 3 Gah; I meant Python 2 here.

(Must proofread my screeds before posting them)

On Thu, Mar 03, 2011 at 09:11:40PM -0500, Barry Warsaw wrote:

On Mar 03, 2011, at 02:17 PM, David Malcolm wrote:

...

On a related note, we have a number of scripts packaged across the distributions with a shebang line that reads: #!/usr/bin/env python which AIUI follows upstream recommendations.

Actually, I think this is *not* a good idea for distro provided scripts. For any Python scripts released by the distro, you know exactly which Python it should run on, so it's better to hard code it. That way, if someone installs Python from source, or installs an experimental version of a new distro Python, it won't break their system. Yes, this has happened to me. Also, note that distutils/setuptools/distribute rewrite the shebang line when they install scripts.

...

There was a proposal to change these when packaging them to hardcode the specific python binary:

https://fedoraproject.org/wiki/Features/SystemPythonExecutablesUseSystemPyth... on the grounds that a packaged system script is expecting (and has been tested against) a specific python build.

That proposal has not yet been carried out. Ideally if we did this, we'd implement it as a postprocessing phase within "rpmbuild", rather than manually patching hundreds of files.

Note that this would only cover shebang lines at the tops of scripts.

JFDI!

FWIW, a quick grep reveals about two dozen such scripts in /usr/bin on Ubuntu. We should fix these. ;)

Note, we were unable to pass Guideline changes to do this in Fedora. Gory details of the FPC meeting are at 16:15:03 (abadger1999 == me): http://meetbot.fedoraproject.org/fedora-meeting/2009-08-19/fedora-meeting.20... The mailing list thread where this was discussed is here: http://lists.fedoraproject.org/pipermail/packaging/2009-July/006248.html Note to dmalcolm: IIRC, that also means that the Feature page you point to isn't going to happen either. Barry -- if other distros adopted stronger policies, then that might justify me taking this back to the Packaging Committee. -Toshio

PEP: ??? Title: The python Utility on Unix-Like Systems Version: ??? Last-Modified: ??? Author: Kerrick Staley <mail at kerrickstaley.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Mar-2011 Post-History: ??? Abstract ========== This PEP provides a convention to ensure that Python scripts can continue to be portable across *nix systems, regardless of the default version of the Python interpreter (i.e. the version invoked by the "python" utility). Recommendation ================ * ``*nix`` software distributions should install the "python2" utility into the default path whenever a version of the Python 2 interpreter is installed, and the same for "python3" and the Python 3 interpreter. When invoked, "python2" should run some version of the Python 2 interpreter, and "python3" should run some version of the Python 3 interpreter. The same applies for the more general "python" command, which should should be installed whenever any version of Python is installed and should invoke some Python interpreter. * All new code that needs to invoke the Python interpreter should not specify "python", but rather should specify either "python2" or "python3" (or the more specific "python2.X" and "python3.X" versions; see the Notes). This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context. Rationale =========== This is needed because some distributions alias the "python" command to Python 3, while others alias it to Python 2. Some of the latter also do not provide a "python2" command; hence, there is no way for Python 2 code (or any code that invokes the Python 2 interpreter) to reliably run on all systems without modification, because both the "python" and the "python2" commands will fail on some systems. The recommendations in this PEP provide a very simple mechanism to restore cross-platform support, with very little additional work required on the part of distribution maintainers. Notes ======= * Distributions can alias the "python" command to whichever version of the Python interpreter they choose. * The "pythonX.X" (e.g. "python2.6") utilities exist on some systems, on which they invoke specific minor versions of the Python interpreter. It would be wise for distribution-specific packages to take advantage of these utilities if they exist, since it will prevent code breakage if the default minor version of a given major version is changed. However, scripts intending to be cross-platform should not rely on the presence of these utilities, but rather should be tested on several recent minor versions of the target major version, compensating, if necessary, for the small differences that exist between minor versions. This prevents the need for sysadmins to install many very similar versions of the interpreter. * It would be wise for distribution-specific packages to always follow the "python2"/"python3" convention, even in code that is not intended to operate on other distributions. This will prevent problems if the distribution decides to upgrade the version of the Python interpreter that the "python" command invokes, or if the sysadmin installs a custom "python" utility with a different major version than the distribution default. Distributions can test whether they are fully following this convention by changing the "python" interpreter on a test box and checking to see if anything breaks. * If the above point is adhered to and sysadmins are permitted to change the "python" utility, then the "python" utility should always be implemented as link to the interpreter binary (or a link to a link) and not vice versa. That way, if the sysadmin does decide to replace the installed "python" file, he can do so without inadvertently deleting the previously installed binary. * The first recommendation can be ignored for systems on which the "python" command itself has traditionally been left undefined and users have always had the responsibility of linking the "python" command to the Python interpreter. * If the Python 2 interpreter becomes uncommon, scripts should nevertheless continue to use the "python3" convention rather that just "python". This will ease transition in the event that yet another major version of Python is released. * If these conventions are adhered to, it will be the case that the "python" utility is only executed in an interactive manner. Backwards Compatibility ========================= A potential problem can arise if a script adhering to the "python2"/"python3" convention is executed on a system not supporting these commands. This is mostly a non-issue, since the sysadmin can simply create these symbolic links and avoid further problems. Copyright =========== This document has been placed in the public domain.

On Fri, 2011-03-04 at 00:54 -0800, Aaron DeVore wrote:

On Thu, Mar 3, 2011 at 11:44 PM, Kerrick Staley <mail@kerrickstaley.com> wrote:

...

That way, if the sysadmin does decide to replace the installed "python" file, he can do so without inadvertently deleting the previously installed binary.

Nit pick: Change "he" to "they" to be gender neutral.

Nit pick: Change "they" to "he" to be grammatically correct. If we really have to be gender neutral, change "he" to "he or she".

On Fri, 4 Mar 2011, R. David Murray wrote:

...

...

Nit pick: Change "he" to "they" to be gender neutral.

Nit pick: Change "they" to "he" to be grammatically correct. If we really have to be gender neutral, change "he" to "he or she".

English is evolving. I vote for "they".

Sorry, can't resist a further nitpick: English has not been evolving in this particular case, in the sense that "they" has been used as a singular since before English was English: http://motivatedgrammar.wordpress.com/2009/09/10/singular-they-and-the-many-... Also interesting: http://en.wikipedia.org/wiki/Singular_they Attention "he or she"ists: Singular "they" won before any of us was born. You may want to divert your energies to a more worthy cause, such as ensuring proper use of "whom". Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist

On Sat, 2011-03-05 at 03:27 +1100, Steven D'Aprano wrote:

Westley Martínez wrote:

...

On Fri, 2011-03-04 at 00:54 -0800, Aaron DeVore wrote:

...

On Thu, Mar 3, 2011 at 11:44 PM, Kerrick Staley <mail@kerrickstaley.com> wrote:

...

That way, if the sysadmin does decide to replace the installed "python" file, he can do so without inadvertently deleting the previously installed binary. Nit pick: Change "he" to "they" to be gender neutral.

Nit pick: Change "they" to "he" to be grammatically correct. If we really have to be gender neutral, change "he" to "he or she".

Actually, that's a hyper-correction imposed by grammarians in the 18th century who were overly influenced by Latin. The use of "they" as a generic singular and indeterminate pronoun in written English goes back at least to Chaucer in the 14th century, and very likely as long back as before English was English, and remains in common use today. Despite the distaste for it among (mostly American) grammarians, it is linguistically sound and widely accepted in most of the English-speaking world, particularly England itself. The use of singular they is widespread, natural, and grammatically correct.

http://www.worldwidewords.org/qa/qa-the2.htm http://en.wikipedia.org/wiki/Singular_they http://itre.cis.upenn.edu/~myl/languagelog/archivs/005423.html

But for the sake of not upsetting our nuclear-armed cousins on the wrong side of the Atlantic *wink*, perhaps the sentence could be reworded to refer to system administrators plural, and thus satisfy everyone?

"That way, if sysadmins decide to replace the installed "python" file, they can do so without inadvertently deleting the previously installed binary."

All right I have to reply to all these "singular they" remarks. Just because the singular they has been used for a long time doesn't make it right. It sounds unnatural, at least to me, and I've always been taught to use "he or she" which I despise. So all my life I've used the generic "he". Anyways, I remember reading somewhere that for Python Strunk and White apply, and neither Strunk nor White like singular theys.

On 3/4/2011 7:40 PM, Guido van Rossum wrote:

On Fri, Mar 4, 2011 at 4:10 PM, Westley Martínez<anikom15@gmail.com>

...

All right I have to reply to all these "singular they" remarks. Just because the singular they has been used for a long time doesn't make it right. It sounds unnatural, at least to me, and I've always been taught to use "he or she" which I despise. So all my life I've used the generic "he". Anyways, I remember reading somewhere that for Python Strunk and White apply, and neither Strunk nor White like singular theys.

That's how I felt 20 years ago.

I still feel that way ...

But since then I've come to appreciate they as a much better alternative to either "he or she" or "he". Just get used to it.

But I discovered long ago that abstract singulars can nearly always be pluralized, as the statement is nearly always about people rather than any particular person. -- Terry Jan Reedy

On Fri, Mar 4, 2011 at 5:44 PM, Kerrick Staley <mail@kerrickstaley.com> wrote:

PEP: ??? Title: The python Utility on Unix-Like Systems

With a few adjustments (formatting, additional info, correction of typos), I've now added Kerrick's PEP as a proposal on python.org: http://www.python.org/dev/peps/pep-0394 The full text is included below as well. Cheers, Nick. PEP: 394 Title: The "python" command on Unix-Like Systems Version: $Revision: 88743 $ Last-Modified: $Date: 2011-03-04 22:04:22 +1000 (Fri, 04 Mar 2011) $ Author: Kerrick Staley <mail at kerrickstaley.com>, Nick Coghlan <ncoghlan at gmail.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Mar-2011 Post-History: 04-Mar-2011 Abstract ======== This PEP provides a convention to ensure that Python scripts can continue to be portable across ``*nix`` systems, regardless of the default version of the Python interpreter (i.e. the version invoked by the ``python`` command). * ``python2`` will refer to some version of Python 2.x * ``python3`` will refer to some version of Python 3.x * ``python`` may refer to either, depending on distribution and system Recommendation ============== * ``*nix`` software distributions should install the ``python2`` command into the default path whenever a version of the Python 2 interpreter is installed, and the same for ``python3`` and the Python 3 interpreter. When invoked, ``python2`` should run some version of the Python 2 interpreter, and ``python3`` should run some version of the Python 3 interpreter. The same applies for the more general ``python`` command, which should be installed whenever any version of Python is installed and should invoke some Python interpreter. * All new code that needs to invoke the Python interpreter should not specify ``python``, but rather should specify either ``python2`` or ``python3`` (or the more specific ``python2.x`` and ``python3.x`` versions; see the Notes). This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context. Note that, when reinvoking the interpreter from a Python script, querying ``sys.executable`` remains the preferred approach. Rationale ========= This is needed as, even though the majority of distributions still alias the ``python`` command to Python 2, some now alias it to Python 3. Some of the former also do not provide a ``python2`` command; hence, there is currently no way for Python 2 code (or any code that invokes the Python 2 interpreter) to reliably run on all systems without modification, because both the ``python`` and the ``python2`` commands will fail on some systems. The recommendations in this PEP provide a very simple mechanism to restore cross-platform support, with minimal additional work required on the part of distribution maintainers. Notes ===== * Distributions can alias the ``python`` command to whichever version of the Python interpreter they choose (noting that, in the near term, most 3rd party scripts will still expect this command to refer to Python 2.x). * The ``pythonX.X`` (e.g. ``python2.6``) utilities exist on some systems, on which they invoke specific minor versions of the Python interpreter. It would be wise for distribution-specific packages to take advantage of these utilities if they exist, since it will prevent code breakage if the default minor version of a given major version is changed. However, scripts intending to be cross-platform should not rely on the presence of these utilities, but rather should be tested on several recent minor versions of the target major version, compensating, if necessary, for the small differences that exist between minor versions. This prevents the need for sysadmins to install many very similar versions of the interpreter. * It would be wise for distribution-specific packages to always follow the ``python2``/``python3`` convention, even in code that is not intended to operate on other distributions. This will prevent problems if the distribution later decides to upgrade the version of the Python interpreter that the ``python`` command invokes, or if a sysadmin installs a custom ``python`` command with a different major version than the distribution default. Distributions can test whether they are fully following this convention by changing the ``python`` interpreter on a test box and checking to see if anything breaks. * If the above point is adhered to and sysadmins are permitted to change the ``python`` command, then the ``python`` command should always be implemented as a link to the interpreter binary (or a link to a link) and not vice versa. That way, if a sysadmin does decide to replace the installed ``python`` file, they can do so without inadvertently deleting the previously installed binary. * The first recommendation can be ignored for systems on which the ``python`` command itself has traditionally been left undefined and users have always had the responsibility of linking the ``python`` command to the Python interpreter. * If the Python 2 interpreter becomes uncommon, scripts should nevertheless continue to use the ``python3`` convention rather that just ``python``. This will ease transition in the event that yet another major version of Python is released. * If these conventions are adhered to, it will be the case that the ``python`` command is only executed in an interactive manner. Backwards Compatibility ========================= A potential problem can arise if a script adhering to the ``python2``/``python3`` convention is executed on a system not supporting these commands. This is mostly a non-issue, since the sysadmin can simply create these symbolic links and avoid further problems. Application to the CPython Reference Interpreter ================================================ While technically a new feature, the ``make install`` command of the 2.7 version of CPython will be adjusted to create the ``python2`` symlink in addition to the existing ``python`` symlink. This feature will first appear in CPython 2.7.2. The ``make install`` command in the CPython 3.x series will continue to install only the ``python3`` symlink for the foreseeable future. Copyright =========== This document has been placed in the public domain. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Fri, Mar 4, 2011 at 10:59 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:

Should any of this also apply to Mac OS X and Windows?

Any platform that considers itself "unix-like" in this context can decide to follow it, we aren't fussy (e.g. Cygwin and the *nix-y aspects of OS X). The main point of the PEP is to get a consensus recommendation out of python-dev as to the best way forward (and I think Kerrick did a good job of summarising the position that has been expressed in this thread). More generally, Windows and Mac OS X developers seem to be happier with the idea of bundling a Python interpreter inside the application than traditional *nix style platforms. This is a PITA for the system maintainer when it comes time to handle security vulnerabilites, but certainly more convenient when upgrading the default Python install.

Note that we *do* have alternative distributors [1] of Python for these platforms who may wish to follow any recommendations we have for 2.7, even if we don't modify those installers for our own distributions.

The really tricky part on Windows is handling file associations. I think we're just doomed on that front, unless we want to start supporting separate .py2 and .py3 extensions (and adding *that* in a maintenance release would be a far cry from just adding another symlink). The lack of near-universal symlink support on Windows filesystems is also an issue - we would have to duplicate files like python.exe and pythonw.exe on non-NTFS filesystems in order to provide them under alternative names. For *nix, I think there is a simple way forward that is an improvement over where things stand now. For Windows, I don't think we can do much better than the status quo and for Mac OS X... I think Apple will do whatever Apple feel like doing :) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

...

...

Should any of this also apply to Mac OS X and Windows? Any platform that considers itself "unix-like" in this context can decide to follow it, we aren't fussy (e.g. Cygwin and the *nix-y aspects of OS X). The main point of the PEP is to get a consensus recommendation out of python-dev as to the best way forward (and I think Kerrick did a good job of summarising the position that has been expressed in this thread).

Right, but the pep doesn't address those issues for some fairly major platforms.

Correct. IIUC (i.e. from the viewpoint of the PEP author), that's deliberate. He just doesn't care about Windows, and, as a PEP author, he clearly has the right to do so :-)

...

More generally, Windows and Mac OS X developers seem to be happier with the idea of bundling a Python interpreter inside the application than traditional *nix style platforms. This is a PITA for the system maintainer when it comes time to handle security vulnerabilites, but certainly more convenient when upgrading the default Python install.

However on Mac OS X at least *scripts* have the same issue (what to put in the shebang line).

IMO, MacOS does fall under the rule of the PEP - it is unixish. I agree with all things that Nick said, in particular with the observation that Apple will do what Apple will do, no matter how many PEPs we write. Our own "make install", for OSX, should clearly do something similar to what it does on other Unix systems.

On Windows we only have a "python.exe" I believe, but if the user does put their Python installs on the path then we *could* usefully create "python2.exe" and "python3.exe" for them. I don't see that duplicating these binaries on the filesystem is an issue. File associations is just unsolvable on Windows, so it isn't something we can address or should worry about. (Actually a stub python.exe that looks at the shebang line and then delegates to the appropriate pythonX.Y.exe would be a possibility but I'm not volunteering to write it.)

Here I also agree with Nick: the binary file name might be the least issue. We had one bug report about 2.x and 3.x interference so far that was difficult to track down - it seems that many Python users on Windows set PYTHONPATH, and that is going to break Python 3 installations. With that problem, and the extension problem, solving the exe name issue doesn't actually resolve real problems. Regards, Martin

Right, but on Mac OS X we do put a "python3" on the path but not a "python2". We also create "python2.x" and "python3.x" variants.

On Windows we only have a "python.exe" I believe, but if the user does put

The PEP makes a recommendation for all *nix platform, which includes Mac OS X. I was not aware that Apple preinstalled Python on OS X, but it doesn't really matter: Apple is the "distribution maintainer", and they will be expected to follow the recommendations of this PEP. Even if Apple is sluggish in getting this change pushed out, it can be resolved on a per-system basis by the sysadmin. their Python

installs on the path then we *could* usefully create "python2.exe" and "python3.exe" for them. I don't see that duplicating these binaries on the filesystem is an issue. File associations is just unsolvable on Windows, so it isn't something we can address or should worry about. (Actually a stub python.exe that looks at the shebang line and then delegates to the appropriate pythonX.Y.exe would be a possibility but I'm not volunteering to write it.)

I like your idea for Windows, but it would take time to implement this solution, and we won't be able to finalize the solution for *nix as quickly if we also provide a provision for Windows in this same PEP. We should keep the use of the singular "they"; it's more popular than the universal "he" (I intended the universal, rather than gender-specific, meaning in the drafts of the PEP). -Kerrick Staley On Fri, Mar 4, 2011 at 9:50 AM, Ronald Oussoren <ronaldoussoren@mac.com>wrote:

On 04 Mar, 2011,at 02:21 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

For *nix, I think there is a simple way forward that is an improvement over where things stand now. For Windows, I don't think we can do much better than the status quo and for Mac OS X... I think Apple will do whatever Apple feel like doing :)

Apple will generally follow what we decide to do for the base install.

Anyway, I'd say that OSX should do the same as Unix platforms here and support '#!/usr/bin/env python2'. Adding another symlink is fairly trivial.

Ronald

P.S. I'm a bit confused about this discussion though, wouldn't adding python2 to the installation be a feature change and as such not something that can be done in a maintenance branch?

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/mail%40kerrickstaley.com

I don't think duplicating python.exe as python2.exe or python3.exe would be very much work at all, if we decide it is a good thing. Sure it doesn't resolve all the myriad problems of Python on Windows but I don't think that is a good reason not to consider it. Up to Martin on this one though and again depends if we just make recommendations or actually change Python 2.7.

Changing the installer should be easy - there is a DuplicateFile table in MSI (*) for this kind of installation task. I'd still like the PEP to tell me whether it's python3w.exe or pythonw3.exe (and yes, that's bikeshedding - so somebody just tell me). It would also be good if the PEP took a position on providing pythonXY.exe binaries on Windows (with the related question of whether it's python32w.exe, python3.2w.exe, pythonw32.exe or pythonw3.2.exe) Regards, Martin (*) http://msdn.microsoft.com/en-us/library/aa368335(v=vs.85).aspx

...

I don't think duplicating python.exe as python2.exe or python3.exe would be very much work at all, if we decide it is a good thing. Sure it doesn't resolve all the myriad problems of Python on Windows but I don't think that is a good reason not to consider it. Up to Martin on this one though and again depends if we just make recommendations or actually change Python 2.7. Changing the installer should be easy - there is a DuplicateFile table in MSI (*) for this kind of installation task.

I'd still like the PEP to tell me whether it's python3w.exe or pythonw3.exe (and yes, that's bikeshedding - so somebody just tell me).

On 04/03/2011 21:35, "Martin v. Löwis" wrote: python3w.exe

It would also be good if the PEP took a position on providing pythonXY.exe binaries on Windows (with the related question of whether it's python32w.exe, python3.2w.exe, pythonw32.exe or pythonw3.2.exe)

I would really like to see this. We do it on other platforms, it is useful there, it would be useful on Windows for the same reasons. All the best, Michael Foord

Regards, Martin

(*) http://msdn.microsoft.com/en-us/library/aa368335(v=vs.85).aspx

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 3/5/2011 12:44 PM, Paul Moore wrote:

On 5 March 2011 15:09, Michael Foord<fuzzyman@voidspace.org.uk> wrote:

...

On 04/03/2011 21:35, "Martin v. Löwis" wrote:

...

It would also be good if the PEP took a position on providing pythonXY.exe binaries on Windows (with the related question of whether it's python32w.exe, python3.2w.exe, pythonw32.exe or pythonw3.2.exe)

I would really like to see this. We do it on other platforms, it is useful there, it would be useful on Windows for the same reasons.

+1. And I'm OK with python3w and python32w as a colour for the bikeshed...

Except for the the spelling of color ;-) I agree. Let 'w' always be the last suffix. With that settled, there is the issue of Start menu shortcuts. I thought we had agreed to put version specific labels on them so we would not have, for instance, identical 'IDLE (Python GUI)' items in the frequently used list. I guess that got lost without a PEP to put it on. Now there is? -- Terry Jan Reedy

...

With that settled, there is the issue of Start menu shortcuts. I thought we had agreed to put version specific labels on them so we would not have, for instance, identical 'IDLE (Python GUI)' items in the frequently used list. I guess that got lost without a PEP to put it on. Now there is?

That would be a bug tracker level issue. I know that I'm bad at wording things (not having English as my native language), so I need somebody to tell me every little detail (preferably by means of a patch). The bug tracker is the right place for that. Agreements on python-dev are quickly forgotten. Regards, Martin

P.S. I'm a bit confused about this discussion though, wouldn't adding python2 to the installation be a feature change and as such not something that can be done in a maintenance branch?

Correct. However, IMO, a PEP could propose to break that rule. Having such a proposal may cause rejection of the PEP (or conditional acceptance only if that change is not carried out). If the PEP is accepted, a special case may be special enough to break the rules. In addition, distributions can add all symlinks they like in their packages, even if "make install" doesn't create them. Whether or not *that* is a feature change depends on the distribution policy. For example, for Debian, it would likely be perfectly reasonable to do that in the 2.7 installation, since 2.7 will be a new feature for Debian, anyway. Regards, Martin

On 4 Mar, 2011, at 19:56, R. David Murray wrote:

On Fri, 04 Mar 2011 15:50:01 +0000, Ronald Oussoren <ronaldoussoren@mac.com> wrote:

...

On 04 Mar, 2011,at 02:21 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

For *nix, I think there is a simple way forward that is an improvement over where things stand now. For Windows, I don't think we can do much better than the status quo and for Mac OS X... I think Apple will do whatever Apple feel like doing :)

Apple will generally follow what we decide to do for the base install.

Anyway, I'd say that OSX should do the same as Unix platforms here and support '#!/usr/bin/env python2'. Adding another symlink is fairly trivial.

FYI, Ronald, the text version of your emails looses all quoting information. It would be nice if you could use standard email quoting (leading '>' characters).

Sorry about this, I keep forgetting how crummy the webmail interface of mobileme is in that regard.

...

P.S. I'm a bit confused about this discussion though, wouldn't adding python2 to the installation be a feature change and as such not something that can be done in a maintenance branch?

The purpose of the "no new features" rule is to prevent the situation where a piece of code written for X.X.2 fails to run on X.X.1 because it relies on a feature introduced in X.X.2. In this situation, we *already* have failures because scripts using /usr/bin/python2 that run on one distro won't run on a different distro where that symlink isn't defined. In this case I don't think the "no new features" rule can be applied blindly, because the feature has *already been introduced* by third parties. What we are attempting to do is make it *more* likely that things will work in the future.

This is a new feature for the cannonical distribution, no release on python.org will install a binary named 'python2'. Adding one now will result in a clear change in behaviour: "#!/usr/bin/env python2" will work in 2.7.2 while it won't work with 2.7.1. That it happens to work right now with some python distributions doesn't mean this isn't a new feature, it's just another instance where the linux distribution maintainers think they know better than the developers.

You can argue that having /usr/bin/python2 installed by 2.7.2 means that code written for 2.7.2 that relies on it won't run on a vanilla user-install of 2.7.1 or 2.7. But how likely is that scenario compared to the scenario where a script written for another distro fails to run because /usr/bin/python2 doesn't exist? I think the balance of the argument comes down in favor of making the change, personally.

That depends on the distributions that have /usr/bin/python2 and change /usr/bin/python to be python3. IIRC the discussion only mentioned Arch Linux and Gentoo, which are AFAIK not major distributions. I'm personally unlikely to run into a distribution that has a broken /usr/bin/python anytime soon, most linux code I write runs on enterprise distributions and those aren't known for being frontrunners with changes like this ;-) BTW. I'm +0 on the change, having python2 can be useful and while this is a new feature it doesn't require major changes when backporting scripts to previous patchlevels. Ronald

On Fri, Mar 4, 2011 at 16:04, Glenn Linderman <v+python@g.nevcal.com> wrote:

Sadly, there seems to be strong resistance to the idea of putting the Python install directory on the Windows path, of course, without some additional solutions (python2.exe, python3.exe, etc.), that doesn't help the multi-version install, only the single version install.

FWIW, I plan on spending time on the PATH issue for 3.3. It seems like much of the resistance could be quelled with actual code. It would be _nice_, but harder, and harder to get consensus on, to write a

little "python launcher" (in a compiled language, not Python, as that would double the startup time) to do some grunge on Windows. Some possibilities, not all would be needed. <SNIP>

* Could also add .py24, .py31, or even .py262, .py301, etc., assoc and

ftype, depending on how many versions of Windows Python are installed.

-INF

On Fri, 04 Mar 2011 01:44:00 -0600, Kerrick Staley <mail@kerrickstaley.com> wrote:

* All new code that needs to invoke the Python interpreter should not specify "python", but rather should specify either "python2" or "python3" (or the more specific "python2.X" and "python3.X" versions; see the Notes). This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context.

If one target for this PEP is script implementors who hadn't thought about this issue before, it is probably worth mentioning sys.executable somewhere (a footnote?). But another issue here (and this speaks against the proposal of not shipping a /usr/bin/python link) is that it is quite possible to write a script that will run on either python2 or python3. -- R. David Murray www.bitdance.com

On Sat, Mar 5, 2011 at 1:10 AM, Barry Warsaw <barry@python.org> wrote:

Folks, please stop CC'ing peps@python.org for non-PEP submissions.  They all get held for moderator approval.  I've approved a few of them, but I'm going to start rejecting them (so you get a bounce :) unless the message actually contains a PEP.

Sorry, I didn't even notice that was on the CC list. Don't add the PEP, I already created PEP 394 for it :) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Is there any discussion still going on about the details of the PEP (now PEP 394)? I'm in favor of the general idea. What about Windows? I think it should be the same there if possible. The only thing I note is that the PEP doesn't explicitly state (unless I missed it) that "python" should invoke the same Python binary as either "python2" or "python3" and not some other version of Python {2,3}.x (at least not until python4 is being considered :-). Similarly, python2 and python3 should refer to one of the existing python2.x or python3.x binary and not some other one. -- --Guido van Rossum (python.org/~guido)

On Mar 4, 2011, at 4:21 PM, Martin v. Löwis wrote:

and setting PYTHONPATH will continue to break installations).

Indeed, it's really *quite* unfortunate that the proposal to make python3 use PYTHON3PATH instead of PYTHONPATH was rejected. James

On Fri, Mar 4, 2011 at 1:21 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:

Am 04.03.2011 20:14, schrieb Guido van Rossum:

...

Is there any discussion still going on about the details of the PEP (now PEP 394)? I'm in favor of the general idea. What about Windows? I think it should be the same there if possible.

I think a key issue is whether to change future 2.7 bug fix releases or not. People have been opposed to the very notion of adding this feature in a bug fix release; not sure how they feel if the change is PEP-sanctioned.

Personally I think this is fine -- it 's not like this is affecting language or library compatibility, but doing this from now on emphasizes that this is what we want distros to do. As far as I'm concerned we could go further and add creation of the python2 symlink to the "make install" target for new bugfix releases of older versions too.

As for Windows support: we currently don't install a python3.exe binary, let alone python2.exe or pythonw2.exe (or is that python2w.exe?). I'll adjust the installer if the PEP asks me to. For the reasons discussed, I'm -0 on the change (i.e. double-clicking .py will continue to launch the most-recently installed Python, rather than the "right" one, and setting PYTHONPATH will continue to break installations).

I'm not a Windows user so I'll leave this part to the group of interested developers to reach consensus. -- --Guido van Rossum (python.org/~guido)

On 5/03/2011 8:21 AM, "Martin v. Löwis" wrote: ...

As for Windows support: we currently don't install a python3.exe binary, let alone python2.exe or pythonw2.exe (or is that python2w.exe?). I'll adjust the installer if the PEP asks me to. For the reasons discussed, I'm -0 on the change (i.e. double-clicking .py will continue to launch the most-recently installed Python, rather than the "right" one, and setting PYTHONPATH will continue to break installations).

I agree with the -0 - I simply don't think it will, in practice, make anyone's life much easier. To run python2 and python3 based scripts in the same environment already requires some manual work by the machine owner (both directories will need to be added to the path) so the additional burden of some other steps (eg, .bat files, doskey alaises etc) doesn't seem that great. There is also a small risk of confusion - people may believe python.exe and python2.exe/python3.exe have different purposes and be confused about when to use which. I think this discussion should be divorced from this PEP and taken up with the discussion about the PATH and the "last installed wins" issue Martin mentions - only all of them taken together will "fix" this issue - not that I personally consider it particularly broken - more like "sub-optimal" :) Cheers, Mark

On 05/03/2011 07:02, Nick Coghlan wrote:

On Sat, Mar 5, 2011 at 10:47 AM, Mark Hammond<mhammond@skippinet.com.au> wrote:

...

I think this discussion should be divorced from this PEP and taken up with the discussion about the PATH and the "last installed wins" issue Martin mentions - only all of them taken together will "fix" this issue - not that I personally consider it particularly broken - more like "sub-optimal" :) I updated the draft PEP, explicitly bringing Mac OS X and Cygwin within the scope of the recommendation, and excluding PYTHON* environment variable considerations and Windows-related proposals. The Windows section does include a hit-list that may serve as a useful starting point if someone else felt like starting a Windows specific PEP, though.

The issues are relevant to Windows (there are *other* issues as well but that doesn't mean that the same issue doesn't apply). Guido also said he would like to see Windows addressed. I'll be happy to try and come to some consensus with Martin (and others) on what we *can* do for Windows (as similar to the other platforms as possible would be my goal) and provide appropriate text for the PEP. I don't think it would be good (or necessary) to split this into a separate PEP. PyCon (sprints or language summit) would be a good place to talk about this.

I believe the only remaining decision to be made is whether we actually change the "make install" command and the Mac OS X installers for Python 2.7.2, or leave them alone and tell the distro folks to fix it on their side of the fence. My own vote is a +1 for both, since changing "make install" is fairly easy, and Ronald indicated earlier in the thread that not only is the change to the Mac OS X installer pretty trivial, but that Apple are likely to follow the lead of whatever our default installer does.

Martin has also indicated that making appropriate changes to the Windows installer would not be difficult if we agree that changing the 2.7 maintenance branch in this way is appropriate. All the best, Michael Foord

Full PEP: http://www.python.org/dev/peps/pep-0394/ Diff to previous version: http://svn.python.org/view/peps/trunk/pep-0394.txt?r1=88743&r2=88753

Cheers, Nick.

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On Sun, Mar 6, 2011 at 12:22 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On Sat, Mar 5, 2011 at 11:37 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:

...

I don't think it would be good (or necessary) to split this into a separate PEP. PyCon (sprints or language summit) would be a good place to talk about this.

Sure. With a PEP to record decisions this time, we shouldn't get a repeat of the last naming discussion where

... people forgot precisely which decisions were made and their rationale. Oops :) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On 6/03/2011 12:37 AM, Michael Foord wrote:

On 05/03/2011 07:02, Nick Coghlan wrote:

...

On Sat, Mar 5, 2011 at 10:47 AM, Mark Hammond<mhammond@skippinet.com.au> wrote:

...

I think this discussion should be divorced from this PEP and taken up with the discussion about the PATH and the "last installed wins" issue Martin mentions - only all of them taken together will "fix" this issue - not that I personally consider it particularly broken - more like "sub-optimal" :) I updated the draft PEP, explicitly bringing Mac OS X and Cygwin within the scope of the recommendation, and excluding PYTHON* environment variable considerations and Windows-related proposals. The Windows section does include a hit-list that may serve as a useful starting point if someone else felt like starting a Windows specific PEP, though.

The issues are relevant to Windows (there are *other* issues as well but that doesn't mean that the same issue doesn't apply). Guido also said he would like to see Windows addressed.

What issues specifically? If I look at the current PEP language, it refers to "the default version of the Python interpreter" and "all new code that needs to invoke the Python interpreter". Neither of these apply in any meaningful way to Windows. It could be argued that the latter could include .bat files which use Python but that sounds like a theoretical problem (ie, I haven't heard much noise about that) where PEP394 is addressing an observed practical problem. On Windows it seems to be just about 'python2' and 'python3' doing the correct thing in a default command prompt, which while desirable ignores the broader issues (eg, the file associations and anything else "windowsy" and not driven by the command-prompt)

I'll be happy to try and come to some consensus with Martin (and others) on what we *can* do for Windows (as similar to the other platforms as possible would be my goal) and provide appropriate text for the PEP.

IIUC, the PEP language is referring to links which point to a specific version of Python and that there is no suggestion a 'python3' will live in the Python 3 binary tree. If that is correct and assuming we don't want to investigate using links on Windows, I'd suggest the best analogy to the *nix situation could be simply for the installers to generate "python2.bat" and "python3.bat" files and put them in System32 with a "last installed wins" policy. The fact this doesn't help users installing Python "just for me" is fine too from the POV of matching *nix systems - a user installing a private Python build on *nix also doesn't get the python2 and python3 conveniences, nor the automatic PATH convenience. But in practice I expect this would *not* be OK - which just highlights the risks of trying to generalize a specific observed problem on one OS to others. Cheers, Mark

On 6/03/2011 11:51 PM, Dj Gilcrease wrote:

Why not modify the windows installers to install into C:\Python\X.Y and have the .bat files generated in C:\Python which is what I have been doing manually since py25. I just add C:\Python to the system Path then create/modify the bat files for new versions of python I install.

That sounds like a reasonable scheme people may choose to use - however, it doesn't really address the basic issue - something still needs to add c:\Python to PATH, and the scheme itself doesn't really necessitate the common parent directory - ie, it is fundamentally the same (if less "clean") as the existing "c:\pythonx" with a (eg) "c:\python-cmd" or "c:\python-scripts" directory with the .bat files - especially given the various installations can be located via the registry. ...

Something I have been thinking about recently though is outside the scope of the pep is writing a python.exe, to replace the python.bat, that would try to read the shebang line of the file to send it to the right version of python. Then I just associate py files with the dispatcher exe and everything should work as intended.

But where would such a python.exe live and how would that directory end up on the PATH? On the more general idea though, it could have legs as it would solve the file association issue for files which include the shebang and arrange for the status-quo (or better) for files which don't... But this sounds like a different PEP ;) Cheers, Mark.

Using batch files is a poor idea, IMO, because you have to explicitly call a batch file from another, or the remainder of the caller will not execute. Installing to System32 s also questionable, but if it's just the launchers, it might be okay. From an installer's perspective, it would really help if those files kept consistent component GUIDs and had a Windows version block so it could upgrade consistently if it changes in the future I think Glenn Linderman hit the use cases on the head; I'm unclear why he was against the overhead of a helper executable. The things I would really want solutions for are these: * double click on a script, and have it launch the right python (2 or 3, w or not) * Probably scan for the final python[.\d]+ string and assume it's relevant. * be able to easily invoke python to interpret a script from the command prompt I'd be comfortable with setting associations to a set of thin executable wrappers which examined the #! line to extract a python version. It could fall back to the "default" version of python, which could be defined as the highest installed on the machine, or could be customizable by the machine's administrator. If this wrapper script passes on all command line parameters, it could also be a reasonable way to invoke python from the command line. Is there a good way for the wrapper to know what versions of python are available on Windows? Moving forward we could have a pythonx.y installer set a value in a known registry key, and document how to register an older python with this key. The default value of the key could be the mechanism for setting a default python version. I'm willing to clarify this and/or look into providing patches if it would help; the only potentially sticky point is the contribution agreement, but I wouldn't expect trouble with my employer. Michael

On 3/6/2011 7:07 AM, Michael Urman wrote:

I think Glenn Linderman hit the use cases on the head; I'm unclear why he was against the overhead of a helper executable. The things I would really want solutions for are these: * double click on a script, and have it launch the right python (2 or 3, w or not) * Probably scan for the final python[.\d]+ string and assume it's relevant. * be able to easily invoke python to interpret a script from the command prompt

I'd be comfortable with setting associations to a set of thin executable wrappers which examined the #! line to extract a python version.

I'm only against the overhead of a helper written in Python, since it would have to launch Python (some explicit version) to run the helper script, and then launch the "right" version of Python to execute the real script. You mention a thin executable wrapper, and I have no problem with the overhead of that, probably. Seems that the PEP addresses two sub problems on Unix created by the coexistence of multiple incompatible major versions of Python: (1) how to invoke the right version of the interactive Python from the command line, and (2) how to specify the right version of Python inside Python scripts. One solution solves both problems on Unix... the declaration that /usr/bin/python (generally on the PATH and invocable from the shell as python) may be an installation dependent version, and the creation of appropriate version specific links and/or binaries for specific major and/or minor versions, to be used either from the command line or the #! line of scripts. For Windows, it is true at present that neither of the above problems has a solution: (1) a fully qualified installation path name must be used to invoke any version of Python, since it is not on the PATH (2) Only the last installed version of Python will be invoked by launching a Python script name something-or-another.py Hence, omitting Windows behaviors from the PEP leaves Windows without a solution for either problem, no worse off than it was before, but neither is it better off. If the PEP wants to address (1) interactive launching from the command line on Windows, that could be done by placing a batch file in System32; providing versioned symlinks in installed on a file system that supports them, or providing versioned binaries on a file system that doesn't support symlinks, either in System32, or by adding the Python installation directory to the PATH. For interactive use, most of these solutions are roughly equivalent in function. Since Windows does not use a #! line, then the solutions for (2) must be different. The ones that I can think of are: (A) declare the Python version in the name of the script file. By doing so in the extension, additional Windows associations could be created to launch different versions of Python. Switching versions would be as simple as renaming the file, changing the extension to include a different version. However, various tools would have to learn about additional extensions that mean Python (syntax directed editors, etc.) And it would be hard to have a script in a module, unless the importer recognized all those extensions too. So this "solution" has ripple effects that make it unattractive, although it is simple to implement the basic feature. (B) declare the Python version in the content of the script file. This cures most of the ripple effects of the above, but requires a "launcher" or "wrapper" program to be designed, implemented, and installed. There are a variety of subsolutions for different ways of declaring the version. (B1) Use the Unix #! line, and attempt to parse the version from it. This assumes there is a Unix #! line, note that Windows-only scripts wouldn't need them in the first place. Looking at and interpreting the Unix #! line, and translating it into an invocation of Python on Windows is clever and doable, but very tightly tied to Python. (B2) Use a second #! line, and attempt to parse a whole command from it, using Windows pathname syntax. This is attractive for cross-platform script, and builds on the Unix #! line which is well-understood by Unix and cross-platform developers. Again, Windows-only scripts wouldn't need a Unix #! line, but if this solution uses a second one, then the first must be created, possibly empty. Also, if there are parameters the script needs that only apply to one platform, they could be placed in the appropriate platform's #! line. (B3) Invent a variant syntax for #! -- perhaps #@. Since it is different than #!, it could be on the first line, if no Unix #! line is needed, but if not found on the first line, the second line would be examined, to allow #! on the first line for a cross-platform script. Same costs/benefits as (B2). (B4) As a variation on (B2) or (B3), the declaration of the version would not have to be a command line format... it could be "#@ Version 3.2", as it is purely up to the launcher/wrapper program to interpret it. This complicates the wrapper script, as it would have to examine the registry to find the particular location of Python, and tightly binds it to Python usage only. I like (B3) best, now that I've thought of it, although I implemented a form of (B2) in a private copy of http.server so I could have a cross-platform CGI script. (B2) and (B3) also allow the wrapper to be configured to work for similar things for other scripting languages, which also have cross-platform problems similar to Python. All the user would have to do is put the extra #!/#@ line in, and associate that extension with the wrapper. By default, for Python, the wrapper would get associated only with .py files.

On Mon, Mar 7, 2011 at 3:02 PM, Kerrick Staley <mail@kerrickstaley.com> wrote:

I think that at this point there's been sufficient agreement that the technical issues on Windows are too complex to warrant addressing them in this PEP, and that a separate PEP will be written addressing Windows, so that is what will happen. A new email thread for the Windows topic should be started so that the discussions can be separated, and those uninterested in the Windows question can stop following that discussion without missing Unix-related comments buried in it.

Indeed. Terry spelled out the reasons for a separate Windows specific PEP pretty well, so I'll add a reference to that in the next PEP update. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Mon, Mar 7, 2011 at 3:19 PM, Mark Hammond <skippy.hammond@gmail.com> wrote:

Without putting too much thought into it, I think a simple scheme could work where the path must either be "/usr/bin/python[23]?" or a fully-qualified path to a Python executable.  IIUC, this should allow most scripts to have a shebang line that "does the right thing" on *nix and Windows systems while still offering maximum flexibility on Windows.

Such a scheme should be able to handle "/usr/bin/env python2" and variants as well. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On 07/03/2011 05:19, Mark Hammond wrote:

[snip...]

...

(B) declare the Python version in the content of the script file. This cures most of the ripple effects of the above, but requires a "launcher" or "wrapper" program to be designed, implemented, and installed. There are a variety of subsolutions for different ways of declaring the version.

<snip a few options>

Without putting too much thought into it, I think a simple scheme could work where the path must either be "/usr/bin/python[23]?" or a fully-qualified path to a Python executable. IIUC, this should allow most scripts to have a shebang line that "does the right thing" on *nix and Windows systems while still offering maximum flexibility on Windows.

"/usr/bin/env python" is popular too, and it would be nice to support the new aliases (or binaries) being created by the pep (python2 / python3). Michael

...

By default, for Python, the wrapper would get associated only with .py files.

There would probably also need to be the same for ".pyw", but it would be almost identical to the .py handling.

Cheers,

Mark _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 7/03/2011 1:55 AM, Paul Moore wrote:

On 6 March 2011 02:33, Mark Hammond<skippy.hammond@gmail.com> wrote:

...

IIUC, the PEP language is referring to links which point to a specific version of Python and that there is no suggestion a 'python3' will live in the Python 3 binary tree. If that is correct and assuming we don't want to investigate using links on Windows, I'd suggest the best analogy to the *nix situation could be simply for the installers to generate "python2.bat" and "python3.bat" files and put them in System32 with a "last installed wins" policy.

No, no, no! Please no! Bat files are a lousy way of writing wrappers on Windows, as they don't nest. I've lost count of the number of times I've been tripped up by putting a series of commands into a batch file, only to have them fail because somewhere in the middle is a bat file wrapper which makes the "outer" bat file stop part way through :-(

To be clear, I was suggesting that using .bat files in system32 is a close analogy to the *nix situation - I didn't mean to advocate for it to actually happen :) Further, I see the creation of a python3.exe in the Python install directory as quite different than the *nix situation (where the 'python3' link is not in the install dir, but instead in a 'system' dir). IOW, I was trying to point out the solution to the problem on *nix doesn't translate that well to Windows, so Windows should not be considered as part of this PEP. That isn't to suggest Windows should not be considered at all. That said though, I'm only -0 on python2.exe/python3.exe - I don't think it will hurt, but also don't think it will help that much in practice. It may also turn out to be unnecessary should a "complete" solution be implemented - eg, a "python launcher" which (a) read the shebang lines and (b) allowed something like "python -3" on the command-line would render both python3.exe and requests to have multiple installed Python versions on the PATH redundant. Cheers, Mark

On 7 March 2011 01:18, Mark Hammond <skippy.hammond@gmail.com> wrote:

That said though, I'm only -0 on python2.exe/python3.exe - I don't think it will hurt, but also don't think it will help that much in practice. It may also turn out to be unnecessary should a "complete" solution be implemented - eg, a "python launcher" which (a) read the shebang lines and (b) allowed something like "python -3" on the command-line would render both python3.exe and requests to have multiple installed Python versions on the PATH redundant.

That sounds like a fairly cool idea. So if I follow what you're suggesting, we'd have a single python.exe, probably installed in system32, which did the necessary command line juggling and shebang parsing, then simply redirected to the appropriate Python interpreter? Presumably that launcher would be pretty version independent, so (a) the one that gets installed with Python 3.3 would support older versions even though they didn't include the launcher themselves, (b) overwriting the launcher when a new version of Python is installed wouldn't be too big a deal, and (c) it could be released as a standalone package for people with only older versions of Python installed? I like this idea. If I had the spare time (I don't :-() I'd work on this myself. +1 from me. Paul.

On 07/03/2011 10:49, Mark Hammond wrote:

On 7/03/2011 9:33 PM, Paul Moore wrote:

...

That sounds like a fairly cool idea. So if I follow what you're suggesting, we'd have a single python.exe, probably installed in system32, which did the necessary command line juggling and shebang parsing, then simply redirected to the appropriate Python interpreter? Presumably that launcher would be pretty version independent, so (a) the one that gets installed with Python 3.3 would support older versions even though they didn't include the launcher themselves, (b) overwriting the launcher when a new version of Python is installed wouldn't be too big a deal, and (c) it could be released as a standalone package for people with only older versions of Python installed?

Yup - although I think a pythonw.exe launcher would be needed too (for the same reasons we need python.exe and pythonw.exe today)

...

I like this idea. If I had the spare time (I don't :-() I'd work on this myself.

+1 from me.

Agreed all round. In the highly unlikely event that I find some time I too might have a play with the idea. The devil will undoubtedly be in the details. I've implemented a Pure-python version of this before, but found it unacceptably slow for anything but ad-hoc use. Still, it was useful for that :) TJG

Mark Hammond wrote:

Yup - although I think a pythonw.exe launcher would be needed too

Couldn't the launcher look at the extension of the file being launched to decide about this? -- Greg

On 07/03/2011 01:18, Mark Hammond wrote:

[snip...] That said though, I'm only -0 on python2.exe/python3.exe - I don't think it will hurt, but also don't think it will help that much in practice. It may also turn out to be unnecessary should a "complete" solution be implemented - eg, a "python launcher" which (a) read the shebang lines and (b) allowed something like "python -3" on the command-line would render both python3.exe and requests to have multiple installed Python versions on the PATH redundant.

A python launcher as you describe is a *great* idea. A few concerns: * we're missing an opportunity to do something easy (Martin is happy to modify the installer and says it is easy) for something that may or may not happen * will you call it python.exe? will it be installed by the python installer? - I doubt calling it python.exe will fly, but I'm not sure. If so what will you call what is currently 'python.exe'? - if not then "python foo.py" on the command line will *still* not work... - it will still have to be installed on the PATH, but I guess System32 will do for that - users without admin rights will still have to modify their PATH manually and place it somewhere else * we're (yet again) making instructions for running stuff on Windows *different* to other platforms (and making tutorials written for other pythons "not work" in certain ways) * as I work with multiple platforms it would be really nice if the same invocations worked across all of them - whilst I say again that I really like the idea of the launcher it doesn't conflict with the other suggestions (creating multiple binaries) and as you (Mark) say it wouldn't hurt... So why not do both? We could create the extra binaries to bring Python on Windows inline with the unix conventions for command line invocations, and the new launcher can follow on as a nice addition. Note that the discussions about the Python installer adding to the PATH won't be *ended* by the creation of the installer. A typical install on a Unix-like system adds various other utilities to the path that merely adding the top-level of your Python install on Windows still doesn't add. In particular distutils installed scripts go into a subdirectory of your Python install. All the best, Michael Foord

Cheers,

Mark

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On Mar 7, 2011, at 3:49 PM, Paul Moore wrote:

The launcher could also (as per Mark's suggestion) interpret a shebang line in the script, so that scripts could specify their required version without needing a different command,or multiple version-specific extensions.

Note that, on Unix, "python file.py" doesn't choose the correct version of python to run by looking at a shebang, it just runs the version of python installed as "python". Only "./file.py" looks at the shebang (assuming that file is marked executable). Is the proposal to make python.exe do that on windows? That is a rather significant difference from the unix behavior.

On 3/7/2011 2:18 PM, James Y Knight wrote:

On Mar 7, 2011, at 3:49 PM, Paul Moore wrote:

...

...

The launcher could also (as per Mark's suggestion) interpret a shebang line in the script, so that scripts could specify their required version without needing a different command,or multiple version-specific extensions. Note that, on Unix, "python file.py" doesn't choose the correct version of python to run by looking at a shebang, it just runs the version of python installed as "python". Only "./file.py" looks at the shebang (assuming that file is marked executable).

Is the proposal to make python.exe do that on windows? That is a rather significant difference from the unix behavior.

The launcher need not be called "python.exe", and maybe it would be better called #@launcher.exe (or similar, depending on its exact function details). For launching from the command line, if various versions python binaries or batch files are on the PATH, then appropriate explicit python invocation python foo.py # Just like Unix python2 foo.py python3 foo.py should launch the specified version (where python may be somewhat ambiguous) but foo.py # with the help of a launcher this could be just like Unix too would implicitly use the launcher, as would launching it from the GUI. There are two problems here, explicit command-line invocation, and GUI/assoc invocation. A launcher need not, and probably should not, do anything for explicit python invocation, but would only solve the implicit and GUI type of invocations.

On 3/8/2011 12:02 PM, Terry Reedy wrote:

On 3/7/2011 9:31 PM, Reliable Domains wrote:

...

The launcher need not be called "python.exe", and maybe it would be better called #@launcher.exe (or similar, depending on its exact function details).

I do not know that the '#@' part is about, but pygo would be short and expressive.

If my proposal to make a line starting with #@ to be used instead of the Unix #! (#@ could be on the first or second line, to allow cross-platform scripts to use both, and Windows only scripts to not have #!), then #@launcher.exe (and #@launcherw.exe I suppose) would reflect the functionality of the launcher, which need not be tightly tied to Python, if it uses a separate line. But the launcher should probably not be the thing invoked from the command line, only implicitly when running scripts by naming them as the first thing on the command line. I'm of the opinion that attempting to parse a Unix #! line, and intuit what would be the equivalent on Windows is unnecessarily complex and error prone, and assumes that the variant systems are configured using the same guidelines (which the Python community may espouse, but may not be followed by all distributions, sysadmins, or users). That's why I propose a different line for Windows... it is as simple as the long-proven Unix #! line, but imposes no restrictions on or requirements that there be a #! line; it has more flexibility in that it could invoke different versions or provide different options on Unix and Windows if necessary for some environments. #!/usr/bin/env python2.6 -B #@c:\python26\python2.6.exe or #!/usr/bin/python2.5 #@"C:\Program Files (x86)\IronPython 2.6 for .NET 4.0\ipy.exe" Now that I've had this idea, one might want to create other "2nd character" codes after the Unix #! line... one could have #! Unix command processor #@ Windows command processor #$ OS/2 command processor #% Alternate Windows command processor. One could even port it to Unix: #!/usr/bin/#@launcher #@c:\python2.6\python.exe #^/usr/bin/python2.5 #&/usr/bin/mono/IronPython2.6 for .NET 4.0/ipy.exe # I made up the line above, having no knowledge of Mono, but I think you get the idea Choice of command line would be an environment variable, I suppose, that the launcher would look at, or if none, then a system-specific default. It would have to search forward in the file until it finds the appropriate prefix or a line not starting with #, or starting with "# " or "##", at which point it would give up.

On 3/8/2011 8:02 PM, Toshio Kuratomi wrote:

On Tue, Mar 08, 2011 at 06:43:19PM -0800, Glenn Linderman wrote:

...

On 3/8/2011 12:02 PM, Terry Reedy wrote:

On 3/7/2011 9:31 PM, Reliable Domains wrote:

The launcher need not be called "python.exe", and maybe it would be better called #@launcher.exe (or similar, depending on its exact function details).

I do not know that the '#@' part is about, but pygo would be short and expressive.

If my proposal to make a line starting with #@ to be used instead of the Unix #! (#@ could be on the first or second line, to allow cross-platform scripts to use both, and Windows only scripts to not have #!

You'd need to allow for it to be on the third line as well. pep-0263 has already taken the second line if it's in a script that has a Unix shebang.

Thanks for your feedback Toshio... Indeed! I've been using Python 3 and UTF-8 encoding only long enough, I forgot about pep-0263, although I'd been aware of it before... Now I only port back to Python 2 for stuff I have to run on my web host, which runs Python 2.6, and those are all ASCII files by design.

...

), then #@launcher.exe (and # @launcherw.exe I suppose) would reflect the functionality of the launcher, which need not be tightly tied to Python, if it uses a separate line. But the launcher should probably not be the thing invoked from the command line, only implicitly when running scripts by naming them as the first thing on the command line.

I'm of the opinion that attempting to parse a Unix #! line, and intuit what would be the equivalent on Windows is unnecessarily complex and error prone, and assumes that the variant systems are configured using the same guidelines (which the Python community may espouse, but may not be followed by all distributions, sysadmins, or users).

I do not have a Windows system so I don't have a horse in this race but if the argument is to avoid complexity, be careful that your proposed solution isn't more complex than what you're avoiding. ie::

...

Now that I've had this idea, one might want to create other "2nd character" codes after the Unix #! line... one could have

#! Unix command processor #@ Windows command processor #$ OS/2 command processor #% Alternate Windows command processor.

One could even port it to Unix:

#!/usr/bin/#@launcher #@c:\python2.6\python.exe #^/usr/bin/python2.5 #&/usr/bin/mono/IronPython2.6 for .NET 4.0/ipy.exe # I made up the line above, having no knowledge of Mono, but I think you get the idea

Choice of command line would be an environment variable, I suppose, that the launcher would look at, or if none, then a system-specific default. It would have to search forward in the file until it finds the appropriate prefix or a line not starting with #, or starting with "# " or "##", at which point it would give up. For PEP-0263 compliance, modify the terminator to be a line not starting with #, or a line starting with "##". This may cause reduced

Yep, even after I wrote the below, I thought about not sending that part, for exactly that reason. But after slightly investigating the other Python implementations, I realized even more strongly that interpreting the Unix #! line is very limiting: 1) it requires the same version to exist on both Unix and Windows, which may often, but not always, be true. 2) it either hard-codes an alternative path name in the launcher, or hard-codes an algorithm to do lookups in the registry to find the alternative path names. Neither of these are very friendly to alternative Python implementations. 3) people concerned with keeping scripts and modules compatible with alternative implementations might have more than one implementation on their system. So the complexity I'm avoiding is the "hard-coding" sort of complexity. The complexity I propose adds flexibility that the hard-coding would be hard-pressed to achieve. performance for scripts starting with large block comments without having the searched-for execution code... but only in the case where the searched-for execution code is not present.

On 3/8/2011 9:06 PM, Mark Hammond wrote:

On 9/03/2011 1:43 PM, Glenn Linderman wrote:

...

I'm of the opinion that attempting to parse a Unix #! line, and intuit what would be the equivalent on Windows is unnecessarily complex and error prone, and assumes that the variant systems are configured using the same guidelines (which the Python community may espouse, but may not be followed by all distributions, sysadmins, or users). That's why I propose a different line for Windows... it is as simple as the long-proven Unix #! line, but imposes no restrictions on or requirements that there be a #! line; it has more flexibility in that it could invoke different versions or provide different options on Unix and Windows if necessary for some environments.

#!/usr/bin/env python2.6 -B #@c:\python26\python2.6.exe

or

#!/usr/bin/python2.5 #@"C:\Program Files (x86)\IronPython 2.6 for .NET 4.0\ipy.exe"

I don't think that buys us much. A script with "#!/usr/bin/env python" could be distributed with an expectation it will work across various different machines (and possibly even different operating systems). A script with "c:\..." could not be distributed and expected to work reliably anywhere. ie, any script with a #! line (or even a #@ line) with a fully qualified Windows path can only be expected to work on a single machine - so there is no need to support both #! and #@ as the script is not even cross-machine portable, let alone cross-platform portable.

Standard installation paths are accepted by about 99% of the users, so embedding standard installation paths can work well for that batch of users. Of course, Windows changes the standard path periodically, so that it different from versions prior to and including WinXP versus versions after WinXP. And they had no standard before WinNT (or if they did, few followed it). Your comment does point out a possible need for multiple standard installation paths just for different versions of Windows, though :( Also, corporate environments are generally quite standardized, so scripts developed within and for a corporation (or modified for use within a corporation) could quite successfully use such. You do make a good point, though, that Unix-based environments have more standardization in their diversity than Windows does. Back when I wrote Unix code, though, there were plenty of system utility programs that had different default paths on different Unix platforms. /usr/bin/env can cover some of that, but is also somewhat of a security hole.

The only way to expect a #! line to work across machines would be to have a "virtual" path - eg just "python2.6" without any path specifier at all. In that case, I see no problem with reusing the #! from *nix systems and treating "/usr/bin" etc as a "virtual" specifier on Windows. If people find a need on Windows to add a fully-qualified path to this line (whatever the spelling), they are implicitly saying this script works only on the current machine.

IOW, as soon as someone has:

#!/usr/bin/env python2.6 -B #@c:\python26\python2.6.exe

in their script, the script is targeted at exactly 1 specific machine, so why not just reuse the #! syntax? OTOH, if an existing script has:

#!/usr/bin/env python2.6 -B

They are attempting to declare in a portable way that Python 2.6 is necessary - so why force them to add a #@ line to make it work on Windows when the #! line is the only clue Windows needs to make it work automagically?

Your premise that using a standard installation path in #@ restricts the script to be targeted to one machine is fallacious, so the conclusions are also. You also missed the fact that the -B parameter was deemed necessary in the above for the Unix machine(s) of interest, but not needed for Windows, which a single line cannot impart (although that is an admittedly contrived example of potentially different option syntax). And the use of cpython on Unix and IronPython on Windows may be appropriate for some corporate environments. A lot of the Unix discussion mentioned things like "curable by the sysadmin" like in regards to the links... a sysadmin implies a corporate environment, and a locked-down path structure. Of course, /usr/bin/env is already a "launcher" type facility, specific to Unix, to mask variations between systems, and to overcome the fact that Unix itself will not walk the PATH to find a non-fully-qualified name. The #! line is clearly is NOT "the only clue Windows needs to make it work automagically" or we wouldn't be having this discussion at all. And it is not at all clear if a Windows machine contains Jython, IronPython, and Cython which one should be launched by a launcher. You could, of course, argue that python-dev is only concerned with CPython, and the launcher can be specific to CPython, and that argument might carry the day, but the CPython registry lookup necessary to make that happen doesn't help the users of alternate implementations a bit, forcing each of them to implement their own launcher as well... or have they already solved this problem some other way that has not been brought to this discussion? I downloaded/installed IronPython today to get the above path to it, but I didn't find any decent documentation for it to learn this sort of thing... a book reference seemed to claim the most comprehensive documentation, but I'm not going to buy a book just to learn that they haven't solved this issue, even if Michael Foord is one of the authors! Of course if appropriate versioned python executables were placed on the Windows path, then an exact corollary to /usr/bin/env could be used on Windows, also... but there would still be the question of whether to use python2.6.exe or ipy2.6.exe, perhaps. And then there is the "w" issue, too. #!/usr/bin/env python2.6 #@"C:\Program Files\Env.exe" ipy2.6.3.exe in .py files and #!/usr/bin/env python2.6 #@"C:\Program Files\Env.exe" ipy2.6.3w.exe in .pyw files. I have scripts that I run on my web server using #!/usr/bin/python2.6 because they only have that and /usr/bin/python which is 2.4.2. But I want to run them locally for testing with (infrequently) Windows cpython 2.6 and (mostly) Windows cpython 3.2, to prove that they will (yes, certain common functions check sys.version to determine how to do things). So using the Unix #! line certainly doesn't work for that case. If I can have only one default, I would want it to be 3.2 locally, and 2.6 on the server, and to manually override something to run 2.6 locally. Yet all my Pythons are installed in the standard installation directory, and still your Unix hack won't let me do what I need. So no matter what is done, it won't solve world hunger. Probably that's why Windows users have been left high and dry for years in this regard... there is no "system" Python on it, out of the box, so the only users are those that are smart enough to download and install and configure things, and those who use a Python embedded into an application, at the cost of a Python installation per application, because there has been no "system" Python, and because Windows users are not perceived to be capable of, or willing to bother with, downloading and installing various requisite dependencies. The resulting vacuum has been filled by people with different points of view, because there has been no technique implemented, none declared to be "best", and anarchy and variant workarounds have abounded.

On 3/8/2011 10:27 PM, Mark Hammond wrote:

On 9/03/2011 5:05 PM, Glenn Linderman wrote:

...

Standard installation paths are accepted by about 99% of the users, so embedding standard installation paths can work well for that batch of users. Of course, Windows changes the standard path periodically, so that it different from versions prior to and including WinXP versus versions after WinXP. And they had no standard before WinNT (or if they did, few followed it). Your comment does point out a possible need for multiple standard installation paths just for different versions of Windows, though :(

The python installer ignores "Program Files" - as much as anything I'm concerned about the drive letter - IIUC, Python will default to installing on the system drive, which may not be C:

The system drive usually is C:, there is a fair bit of software that won't work when it is not C:, and in any particular corporate environment it is usually standardized by the IT folks. And you are referring specifically to the CPython installer as the IronPython (for example) does install in Program Files.

...

...

Your premise that using a standard installation path in #@ restricts the script to be targeted to one machine is fallacious, so the conclusions are also.

Sorry - I should have said "some machines with Python installed by the installer". Contrast this to the option of looking up the registry and you have "all machines with Python installed by the installer". I'm not going to quibble about the percentages in the first category as I have no data to guide me, but I'm reasonably confident about the "all" in the second.

...

...

The #! line is clearly is NOT "the only clue Windows needs to make it work automagically" or we wouldn't be having this discussion at all.

Sorry - I should have been clear - I meant "the only clue a windows specific Python launcher needs".

Sorrier yet :) I think you really meant "the only clue a Windows specific CPython specific Python launcher needs.

...

And it is not at all clear if a Windows machine contains Jython, IronPython, and Cython which one should be launched by a launcher.

I think it is reasonably clear: it would be the same version which would be launched on most *nix systems. Are there any systems where "/usr/bin/python" would use Jython, for example?

I have no idea the answer to your question... I don't think you do either. There may be some. The sysadmin can change what /usr/bin/python points to. I have seen plenty of Unix systems where /bin/sh is really bash, for example, so if the replacement is "compatible enough" the sysadmin may choose to make such a change.

...

You could, of course, argue that python-dev is only concerned with CPython, and the launcher can be specific to CPython, and that argument might carry the day, but the CPython registry lookup necessary to make that happen doesn't help the users of alternate implementations a bit, forcing each of them to implement their own launcher as well...

I'd have no problem with the Windows launcher supporting other implementations - I expect something like "/usr/bin/jython" would be reasonable. How we locate the installed jython is indeed an interesting problem, but I'd guess it is surmountable - especially if we can get the buy-in of these alternate implementations for their future releases.

Indeed. But now you have a script that require jython on all machines that it runs on, rather than being flexible in being configurable to use the preferred implementations of a particular environment (perhaps CPython 2.6 on Unix boxes, and Jython 2.5 on Windows boxes, or vice versa).

...

#!/usr/bin/env python2.6 #@"C:\Program Files\Env.exe" ipy2.6.3.exe

As above, the C: assumes Windows is installed on C: - that may be common, but not universal. Further, a 64bit machine would probably want "\Program Files (x86)" (which is becoming more and more common)

Yes, same C: issue as above, see response above. And if the launcher comes in a 64-bit version for 64-bit machines, then it would be installed as C:\Program Files\Env.exe on the 64-bit machines. That should be straightforward to achieve.

...

in .py files and

#!/usr/bin/env python2.6 #@"C:\Program Files\Env.exe" ipy2.6.3w.exe

in .pyw files.

I'm slightly skeptical about that - .pyw files are surely rare on *nix systems and as we've discussed, pyw files are generally not meant to be cmd-line scripts. IOW, I'm skeptical scripts exist which are meant to work with '/usr/bin/env python2.6' on *nix and pythonw.exe on windows.

Indeed, .pyw files are even rare on my Windows box :) A GUI sort of Python script would generally want to be .pyw on a Windows box. There are a variety of cross-platform GUIs such as Qt and wxWindows, so those same scripts could be written to work on Unix boxes also. Unix, happily, doesn't care about the name of the script as long as the #! line points the right place, and neither does CPython on Unix, at least for non-module scripts (which are the ones that have #! lines anyway). So it is quite conceivable, even if you have never done it personally.

...

...

So no matter what is done, it won't solve world hunger. Probably that's why Windows users have been left high and dry for years in this regard... there is no "system" Python on it, out of the box, so the only users are those that are smart enough to download and install and configure things, and those who use a Python embedded into an application, at the cost of a Python installation per application, because there has been no "system" Python, and because Windows users are not perceived to be capable of, or willing to bother with, downloading and installing various requisite dependencies.

The resulting vacuum has been filled by people with different points of view, because there has been no technique implemented, none declared to be "best", and anarchy and variant workarounds have abounded.

I'm glad solving world hunger is out of scope for this :) I understand your position but my personal opinion is that simple support for #! is more desirable. I'd be happy to go with the consensus though...

The solution should be as simple as possible, but no simpler. -- Albert I agree that parsing #! for a version and looking that up in the CPython registry entries would help in the limited circumstances where only the CPython implementation is of interest, and the same versions are available with the same names on all the machines of interest to the script. Of course, my web host has /usr/bin/python (CPython 2.4.2), /usr/bin/python2.4 (Cpython 2.4.2) and /usr/bin/python (Cpython 2.6 -- apparently 2.6.0). So when I choose the first, I get whatever the sysadmin chooses for that (they are slow to make changes for compatibility reasons, which is good). When I choose the second for my #! line I get 2.4.2 there. Now if I have all of 2.4, 2.4.1, 2.4.3, and 2.4.4 versions installed on my Windows box, which one should be run in the case of /usr/bin/python2.4? The registry doesn't seem to tell which was installed first, second, or last. Should it pick the newest, or the exact match of 2.4? It can hardly intuit which version is most compatible with the other machines I run it on. Whatever is picked would seem to be some heuristic. And what if there is only 2.5 on the Windows boxes? Or what if there is no CPython on the Windows box, but only Jython or IronPython, etc. the heuristics would have to understand all those mappings between versions... or simply not work at all, with no fallback technique to help it work. With the #@ idea, application installers could ask which version of Python the application should be configured to use, from among the installed versions that the application has been tested with, and update the #@ lines, and that would stay constant for the life the application (and if the application has an installer that is that smart, it also wouldn't need a #! line, likely, when installed for Windows). If there is no installer, then the user (or his sysadmin) must be smart enough to know how to use a text editor, and configure the version he wants by putting its path in the #@ line... and if the user (or his sysadmin) wants to use it cross-platform, then they put in the appropriate #! line for their Unix machines too. Some configuration for the launcher would be required to handle the case where there is no #! line, or else in those cases, the launcher could report "no #@ line" like Unix's shell reports "syntax error" for python scripts handed to /bin/sh because the #! line was omitted.

On 9 March 2011 06:27, Mark Hammond <mhammond@skippinet.com.au> wrote:

I'm glad solving world hunger is out of scope for this :)  I understand your position but my personal opinion is that simple support for #! is more desirable.  I'd be happy to go with the consensus though...

Just in case you need some reassurance about the level of consensus, I agree that simple support for #! is sufficient. Any more is YAGNI. (But if you manage to solve world hunger, do let us know :-)) Paul.

On 8/03/2011 7:33 AM, Michael Foord wrote:

A python launcher as you describe is a *great* idea.

A few concerns:

* we're missing an opportunity to do something easy (Martin is happy to modify the installer and says it is easy) for something that may or may not happen

Don't let my -0 stop anyone :)

* will you call it python.exe? will it be installed by the python installer?

- I doubt calling it python.exe will fly, but I'm not sure. If so what will you call what is currently 'python.exe'? - if not then "python foo.py" on the command line will *still* not work...

Calling it python.exe would make the most sense for people who don't look behind the curtain, but I agree it could potentially be confusing for people. Further, we would need to ensure we didn't create an infinite loop where the launcher python.exe found a python.exe it thought was an appropriate sub-process, but where it turns out it is actually another launcher. Having it installed by the Python installer also makes sense to me but I'd be very interested in Martin's take on this (and also on everything else we are discussing here).

* we're (yet again) making instructions for running stuff on Windows *different* to other platforms (and making tutorials written for other pythons "not work" in certain ways)

Actually, I'd argue we are making them more similar as we would be honoring a shebang line and allowing the exact same mechanism be used to denote the major version of Python needed.

* as I work with multiple platforms it would be really nice if the same invocations worked across all of them - whilst I say again that I really like the idea of the launcher it doesn't conflict with the other suggestions (creating multiple binaries) and as you (Mark) say it wouldn't hurt...

So why not do both? We could create the extra binaries to bring Python on Windows inline with the unix conventions for command line invocations, and the new launcher can follow on as a nice addition.

Note that the discussions about the Python installer adding to the PATH won't be *ended* by the creation of the installer. A typical install on a Unix-like system adds various other utilities to the path that merely adding the top-level of your Python install on Windows still doesn't add. In particular distutils installed scripts go into a subdirectory of your Python install.

The distutils scripts are a good point. To be honest, apart from the "file association" issue, I can't see much advantage in doing both - if all Python directories end up on your path such that "python3 foo.py" magically works, then the launcher script is adding complexity without bringing much to the table. Further, I'm somewhat skeptical that the file associations are used by that many people in the real world - currently when you double-click on a script you get a temp console created, and should a traceback be raised, you get to see it for about 2ms before the console is destroyed. Some people may be .pyw to avoid that, but then their script has to go to extraordinary lengths to display such errors in a UI of some kind. I suspect most people just find it more convenient to launch such scripts from a console. Maybe a quick poll on python-list would be reasonable... Cheers, Mark

Michael Foord wrote:

The launcher program could thrive without *having* to be part of a standard Python install. Offering it separately makes sense even if it *is* included. If we do both then users can vote with their feet.

A launcher might be difficult to integrate into the Python installer, as there can, by definition, only be one. What if I install a new version of Python and then uninstall it? Will the launcher be uninstalled as well? Will it be reverted to a previous version, and if so, which? I am actually working on such a launcher for myself right now. The main problem I am trying to solve is doubleclicking a .py or .pyw file and having it run in the Python version it is intended for. From the command line, I want "something.py arg1 ... argn" to do the right thing as well. I don't care much for fixing "python something.py arg1 ... argn", as I personally rarely need to pass options to Python. I may look into that later, though. My launcher parses the first line of the script for something that starts with "#!" and contains "python<something>". It tries to match that with a list of installed Python versions obtained from the registry, falling back to a configurable default if no such line is found. As to performance of such a solution: I'm writing this in Python (2.5, as its MSVCRT is easier to include in an exe) and converting it to an executable with py2exe. The extra startup time is manageble on my system. Using Python to write this in makes it much easier to experiment with adding features. Once the dust has settled I may attempt to rewrite it in C. Anyway, I believe that such a launcher should not be part of Python itself at first; an external program makes it easier to figure out what works well. As to adding Python to the Windows path: I'd be mildly annoyed if the installer offered that option, and seriously annoyed if it defaulted that option to "yes". Gertjan.

Calling it python.exe would make the most sense for people who don't look behind the curtain, but I agree it could potentially be confusing for people. Further, we would need to ensure we didn't create an infinite loop where the launcher python.exe found a python.exe it thought was an appropriate sub-process, but where it turns out it is actually another launcher.

Having it installed by the Python installer also makes sense to me but I'd be very interested in Martin's take on this (and also on everything else we are discussing here).

I think I would be opposed to adding a launcher to 2.7. It also wouldn't be necessary - if it was released with 3.3, then it could still do version switching for 2.7. If it's called "python.exe", I wonder what it should do when given a file that doesn't carry version information.

I suspect most people just find it more convenient to launch such scripts from a console. Maybe a quick poll on python-list would be reasonable...

I certainly have script files that I double-click. However, those happen to be batch files, not Python. If I would do scripting in Python (which I don't do much these days), I would like to be able to double-click them. I always write my scripts so that they don't give exceptions :-) Actually, the one Python script I run regularly is msi.py, and I currently launch it in a terminal window, because I need to run it with c:\python25\python.exe, which double-clicking won't do for me. If I could double-click it, I would like that more (there is also the issue that the script needs the VS envvars set, so I'd need to find a solution to that, also). Regards, Martin

Michael Foord wrote:

- I doubt calling it python.exe will fly, but I'm not sure. If so what will you call what is currently 'python.exe'? - if not then "python foo.py" on the command line will *still* not work...

However, if it's installed as the exe associated with the .py and .pyw extensions, then simply 'foo.py' on the command line *will* work, and will work better than it does now. -- Greg

On 3/7/2011 4:00 PM, Michael Foord wrote:

On 07/03/2011 23:52, Greg Ewing wrote:

...

Michael Foord wrote:

...

- I doubt calling it python.exe will fly, but I'm not sure. If so what will you call what is currently 'python.exe'? - if not then "python foo.py" on the command line will *still* not work...

However, if it's installed as the exe associated with the .py and .pyw extensions, then simply 'foo.py' on the command line *will* work, and will work better than it does now.

So long as '.py' and '.pyw' are set in the PATHEXT environment variable. (Which again the Python installer doesn't do by default.)

No, PATHEXT only means you can invoke foo.py and foo and get the same results (sometimes, depending on what all in on PATH and PATHEXT)

Sorry that I dropped the ball on this. I'd still like to see this get implemented, but I got distracted with school and forgot about it. Updates I have made to the PEP will be sent as a patch immediately after this email. Here's a summary of what was happenening when we left off: * The draft SVN version from March 4 was pretty much complete. * Some were concerned about addressing Windows, but it was generally agreed to leave the Windows issue to another PEP. PEP 397 was started on March 15 to address the Windows side of the issue. PEP 397 recommends that the Windows Python launcher read the shebang and use it to determine which Python version to use; this allows one syntax for both operating systems that is compatible with the current PEP 394 recommendation. * There were concerns from Ned Deily about the naming of other binaries such as idle, pydoc, and python-config; these need to be created as idle2, pydoc2, and python2-config, with links created at the locations of the original binaries. * There were concerns from Glenn Linderman that the shebang line doesn't encode enough information to flexibly handle Windows launching (or even launching in general). ==== Here are my thoughts: * For Ned's comments, I agree. Although the issue isn't as large with these programs, there's no reason we can't handle them in the same way. I updated the PEP. * For Glenn's comments, I think the method you propose adds too much complexity. Regardless, if the #@ syntax is implemented, it can be described in PEP 397; it won't have an impact on the contents of this PEP. I think, though, that having multiple syntaxes may cause many scripts to be incompatible with multiple platforms when they don't have to be, since Unix coders will rarely add a #@ line, and Windows coders will likely forget the #! line. Also, #@ really ignores the purpose of a shebang: shebangs simply indicate an interpreter that works with the script; the shebang allows users to run arbitrary scripts without worrying about which interpreter they should specify. There's no reason that a script should use one interpreter on Unix, but be incompatible with that interpreter on Windows yet compatible with another. The name of the Unix binary is enough to determine the implementation and version of the interpreter to be used on Unix, and a Windows launcher should always invoke the same implementation/version on Windows (and this won't require hard-coding anything into the launcher if done right). If you want the script to run with a specific interpreter and version, possibly contingent on which operating system you're running the script under, then you can just invoke the interpreter by name with the script as an argument (e.g. python3 myprogram.py). TL;DR: shebangs encode a default implementation/version, and if you need something special, you can just manually run python3 myprogram.py or use a .bat file. ==== Also, I updated the PEP with the clarification that commands like python3 should be hard links (because they'll be invoked from code and are more efficient; also, hard links are just as flexible as symlinks here), while commands like python should be soft links (because this makes it clear to sysadmins that they can be "switched", and it's needed for flexibility if python3 changes). This really doesn't matter, but can we keep it this way unless there are serious objections? Regards, Kerrick Staley

Am 08.03.2011 01:00, schrieb Michael Foord:

On 07/03/2011 23:52, Greg Ewing wrote:

...

Michael Foord wrote:

...

- I doubt calling it python.exe will fly, but I'm not sure. If so what will you call what is currently 'python.exe'? - if not then "python foo.py" on the command line will *still* not work...

However, if it's installed as the exe associated with the .py and .pyw extensions, then simply 'foo.py' on the command line *will* work, and will work better than it does now.

So long as '.py' and '.pyw' are set in the PATHEXT environment variable. (Which again the Python installer doesn't do by default.)

Running foo.py works fine with the current installers (just try to see for yourself). You don't need PATHEXT for that. Regards, Martin

On Sun, Mar 6, 2011 at 11:41, Michael Foord <fuzzyman@voidspace.org.uk>wrote:

I would like to see us create version specific (i.e. python32.exe / python32w.exe) binaries (or links if we drop support for earlier versions of Windows or some filesystems - I'm agnostic on that issue) *plus* a python3.exe / python3w.exe with "last install wins" (as it is for currently for file associations).

I don't have an ActiveState install on this machine, but I know they have been adding some form of version specific binaries for a while, so maybe they can comment on what they chose and how they chose it, and how people are using it. Are Sridhar or Trent on here? (sorry if this was already mentioned -- I joined late and only skimmed many of the posts)

On 07/03/2011 17:21, Sridhar Ratnakumar wrote:

On Sunday, March 6, 2011 at 9:53 AM, Brian Curtin wrote:

...

On Sun, Mar 6, 2011 at 11:41, Michael Foord<fuzzyman@voidspace.org.uk <mailto:fuzzyman@voidspace.org.uk>>wrote: - Hide quoted message -

...

I would like to see us create version specific (i.e. python32.exe / python32w.exe) binaries (or links if we drop support for earlier versions of Windows or some filesystems - I'm agnostic on that issue) *plus* a python3.exe / python3w.exe with "last install wins" (as it is for currently for file associations). I don't have an ActiveState install on this machine, but I know they have been adding some form of version specific binaries for a while, so maybe they can comment on what they chose and how they chose it, and how people are using it. Are Sridhar or Trent on here? ActivePython for Windows includes the following binaries on Windows:

* python.exe * python3.2.exe * python3.xe (plus the `pythonw` versions)

python3.2.exe is better than python32.exe because that matches the invocation (eg: "python3.2 -m foo") on Unix.

Similarly for 2.x (python2.exe / python2.7.exe).

All of this is particularly useful if the installer adds Python directory to %PATH% (which is the case for ActivePython, but not the one from python.org).

Sounds ideal. :-) Michael

-srid

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html