On 03/11/2015 01:38 PM, M.-A. Lemburg wrote:

On 11.03.2015 21:03, David Mertz wrote:

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

+1 -- ~Ethan~

+1 An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs). $ python2.7 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

$ python3.4 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python3.4

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython3 <== #!/usr/bin/python3.4

So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above). I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python". Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package. ~ Ian Lee On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal@egenix.com> wrote:

On 11.03.2015 21:03, David Mertz wrote:

...

https://www.python.org/dev/peps/pep-0394/

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Mar 11 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Sure, and I agree that called "ipython2" / "ipython3" explicitly is better. I just expected it would default to use whatever "python" is implicitly on my path. E.g. that "python" and "ipython" would use the same interpreter. ~ Ian Lee On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert@yahoo.com> wrote:

On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521@gmail.com> wrote:

+1

An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).

Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?

$ python2.7 -m pip install --user ipython

...

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

$ python3.4 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python3.4

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython3 <== #!/usr/bin/python3.4

So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).

I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".

Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.

~ Ian Lee

On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal@egenix.com> wrote:

...

On 11.03.2015 21:03, David Mertz wrote:

...

https://www.python.org/dev/peps/pep-0394/

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Mar 11 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mar 11, 2015, at 6:52 PM, Andrew Barnert <abarnert@yahoo.com> wrote:

On Mar 11, 2015, at 3:45 PM, Ian Lee <ianlee1521@gmail.com <mailto:ianlee1521@gmail.com>> wrote:

...

Sure, and I agree that called "ipython2" / "ipython3" explicitly is better. I just expected it would default to use whatever "python" is implicitly on my path. E.g. that "python" and "ipython" would use the same interpreter.

That's actually not a bad idea; I didn't even think about it because Unix stuff doesn't "naturally" work that way, but it's probably not too hard to do it "unnaturally".

Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython”.

This is a problem that we’ve yet to tackle really (besides special cases for pip itself). You can’t really do it reasonably without not making Wheels, or making wheels that are python version specific. This is because the build process runs on the author’s machine and “bakes” in the names of the commands at that time. It’s likely going to require figuring out which trade off makes the most sense, adding explicit fooX.Y is one option, but that leaves into question what we do with alternate interpreters like PyPy or Jython (including versioning them, because you might have multiple versions of them installed). It also can cause issues when you have multiple things called “python2” or “python2.7” on your path, which version of “python2” does “foo2” call? Obviously it’s possible to just define something and say “this is what it is” but we need to figure out what that thing should be, and if the trade offs make sense. On the other side, executing versioned commands with -m is not quite as nice (python2.7 -m pip is more letters than pip2.7 and doesn’t autocomplete) however it is really explicit under which interpreter it’s going to run and support for things like alternate interpreters and multiple versions is built in.

...

~ Ian Lee

On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert@yahoo.com <mailto:abarnert@yahoo.com>> wrote: On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521@gmail.com <mailto:ianlee1521@gmail.com>> wrote:

...

+1

An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).

Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?

...

$ python2.7 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

$ python3.4 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python3.4

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython3 <== #!/usr/bin/python3.4

So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).

I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".

Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.

~ Ian Lee

On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal@egenix.com <mailto:mal@egenix.com>> wrote: On 11.03.2015 21:03, David Mertz wrote:

...

https://www.python.org/dev/peps/pep-0394/ <https://www.python.org/dev/peps/pep-0394/>

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7.

-- Marc-Andre Lemburg eGenix.com <http://egenix.com/>

Professional Python Services directly from the Source (#1, Mar 11 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ <http://www.egenix.com/> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ <http://zope.egenix.com/> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ <http://python.egenix.com/>

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com <http://egenix.com/> Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ <http://www.egenix.com/company/contact/> _______________________________________________ Python-ideas mailing list Python-ideas@python.org <mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>

_______________________________________________ Python-ideas mailing list Python-ideas@python.org <mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>_______________________________________________

Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert <abarnert@yahoo.com> wrote:

Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython".

hmm -- that seems odd -- if you re-arranged the path after it was installed, it would be different??? I think you can count on folks use PATH to determine what they want as default -- so in python2, setuptools would put ipython and ipython2 and in the python3 install, it would put: ipython and ipython3 so if you asked for ipython3 or ipython2, you'd get what you asked for -- and if you asked for ipython the PATH would determine what you'd get. -CHB

~ Ian Lee On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert@yahoo.com> wrote:

...

On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521@gmail.com> wrote:

+1

An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).

Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?

$ python2.7 -m pip install --user ipython

...

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

$ python3.4 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python3.4

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython3 <== #!/usr/bin/python3.4

So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).

I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".

Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.

~ Ian Lee

On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal@egenix.com> wrote:

...

On 11.03.2015 21:03, David Mertz wrote:

...

https://www.python.org/dev/peps/pep-0394/

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Mar 11 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Mar 11, 2015, at 4:06 PM, Chris Barker <chris.barker@noaa.gov> wrote:

...

On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert <abarnert@yahoo.com> wrote: Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython".

hmm -- that seems odd -- if you re-arranged the path after it was installed, it would be different???

Yes. That's essentially why I said Unix doesn't naturally work that way. But it may be appropriate for this specific use case, even if it's a hack.

I think you can count on folks use PATH to determine what they want as default -- so in python2, setuptools would put ipython and ipython2

and in the python3 install, it would put:

ipython and ipython3

so if you asked for ipython3 or ipython2, you'd get what you asked for -- and if you asked for ipython the PATH would determine what you'd get.

That's exactly what we already have; the problem is when python2 and python3 have the same scripts directory--say, /usr/local/bin--they both install a script called /usr/local/bin/ipython, with whichever was installed last overwriting the one that was installed first. You might argue that this should never come up, but the fact is that it does. When you install python2 and python3 packages from most package managers, they both go into /usr/local, and both install scripts to /usr/local/bin. And on many platforms, even if the system Python is in /usr, it installs scripts to /usr/local/bin (and that's the right thing to do if /usr/bin is only for system binaries or binaries owned by the system package manager). So, Ian (I think it was him; it's hard to see from this pruned reply...) suggested that just "ipython" should be able to run whatever version of Python is just "python" rather than whichever was installed last. How could we make that work? One option is to have the script #!/usr/bin/env python. But, while that works great for the case where you've installed ipython2 and ipython3 and want ipython to follow your python, the case Ian is trying to solve, it doesn't work an all for the case where you've installed only ipython3 and want to be able to access it as just plain ipython, which is an even more common case. (I do have 4 versions of ipython installed for different CPython and PyPy versions, but I only have one chardetect, etc., because I don't care which version of Python runs the script, I just care what the script does. So, the installer hack would solve both problems, on every kind of platform, for every kind of script. But it's a pretty bad hack. Is it worth doing? I'm not sure. But it's worth thinking about why it would get what Ian wants, at least to see why other solutions won't.

-CHB

...

...

~ Ian Lee

...

On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert@yahoo.com> wrote:

...

On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521@gmail.com> wrote:

+1

An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).

Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?

...

...

$ python2.7 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

$ python3.4 -m pip install --user ipython

$ head -n1 ~/.local/bin/ipython* ==> /home/ianlee1521/.local/bin/ipython <== #!/usr/bin/python3.4

==> /home/ianlee1521/.local/bin/ipython2 <== #!/usr/bin/python2.7

==> /home/ianlee1521/.local/bin/ipython3 <== #!/usr/bin/python3.4

So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).

I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".

Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.

~ Ian Lee

...

On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal@egenix.com> wrote: On 11.03.2015 21:03, David Mertz wrote: > https://www.python.org/dev/peps/pep-0394/

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Mar 11 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg@krypto.org> wrote:

I think /usr/bin/env is a bad idea to promote. It changes from an explicit designation of what you need in order to execute to something being derived from the users $PATH. It is *less specific* than /usr/bin/python. For some people that is the goal, but I discourage it.

I don't think the goal is specificity, or lack there of, but a shift in control: Who controls where python is installed? If you put an explicit full path on the #! line, then the script author controls which python is used, AND where it must be installed. If you put /usr/bin env on the #! line, then the script author is saying 'use python", and the script user can have python installed anywhere they want, and can use PATH to tell everything what the default it. I think that _should_ be encouraged -- but ideally with some versioning: #!/usr/bin/env python2 or even: #!/usr/bin/env python3.4 If any of you remember what a pain it was when RedHat installed system tools with (I think): !#/usr/bin/env python You'll know what a nightmare that was -- you could not upgrade the 'default" python without breaking stuff. So system tools should probably use the full specific path -- the script is tied closely to the environment. But if they had at least put a version in there, it would have worked fine: #!/usr/bin/env python1.5 These days, I try to use distutils or setuptools to install scripts, and they can do the right thing to use the correct python to start up anyway... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker <chris.barker@noaa.gov> wrote:

On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg@krypto.org> wrote:

...

I think /usr/bin/env is a bad idea to promote. It changes from an explicit designation of what you need in order to execute to something being derived from the users $PATH. It is *less specific* than /usr/bin/python. For some people that is the goal, but I discourage it.

I don't think the goal is specificity, or lack there of, but a shift in control:

Who controls where python is installed?

If you put an explicit full path on the #! line, then the script author controls which python is used, AND where it must be installed.

If you put /usr/bin env on the #! line, then the script author is saying 'use python", and the script user can have python installed anywhere they want, and can use PATH to tell everything what the default it.

I think that _should_ be encouraged -- but ideally with some versioning:

#!/usr/bin/env python2

or even:

#!/usr/bin/env python3.4

Please, just explicit major versions, not minor ones. python3, not python3.4. Just earlier this year I had to fix about 20 scripts that ran perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks.

If any of you remember what a pain it was when RedHat installed system tools with (I think):

!#/usr/bin/env python

You'll know what a nightmare that was -- you could not upgrade the 'default" python without breaking stuff.

So system tools should probably use the full specific path -- the script is tied closely to the environment. But if they had at least put a version in there, it would have worked fine:

#!/usr/bin/env python1.5

These days, I try to use distutils or setuptools to install scripts, and they can do the right thing to use the correct python to start up anyway...

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

the PEP is pretty perfect. #!/usr/bin/env python_binary_this_script_runs_with if it runs with python 2: #!/usr/bin/env python2 if it runs with python 3: #!/usr/bin/env python3 if it runs with both: #!/usr/bin/env python the only thing worth discussing IMHO is if python 3 should start shipping with *all* symlinks from past 3.x minor versions to the newest minor version. then we could switch to #!/usr/bin/env python3.4 if it uses e.g. pathlib and would be sure that the script will work with future versions. but then again, users can install pathlib for 3.3… ​ Ryan Gonzalez <rymg19@gmail.com> schrieb am Do., 12. März 2015 um 00:13 Uhr:

On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker <chris.barker@noaa.gov> wrote:

...

On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg@krypto.org> wrote:

...

I think /usr/bin/env is a bad idea to promote. It changes from an explicit designation of what you need in order to execute to something being derived from the users $PATH. It is *less specific* than /usr/bin/python. For some people that is the goal, but I discourage it.

I don't think the goal is specificity, or lack there of, but a shift in control:

Who controls where python is installed?

If you put an explicit full path on the #! line, then the script author controls which python is used, AND where it must be installed.

If you put /usr/bin env on the #! line, then the script author is saying 'use python", and the script user can have python installed anywhere they want, and can use PATH to tell everything what the default it.

I think that _should_ be encouraged -- but ideally with some versioning:

#!/usr/bin/env python2

or even:

#!/usr/bin/env python3.4

Please, just explicit major versions, not minor ones. python3, not python3.4. Just earlier this year I had to fix about 20 scripts that ran perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks.

...

If any of you remember what a pain it was when RedHat installed system tools with (I think):

!#/usr/bin/env python

You'll know what a nightmare that was -- you could not upgrade the 'default" python without breaking stuff.

So system tools should probably use the full specific path -- the script is tied closely to the environment. But if they had at least put a version in there, it would have worked fine:

#!/usr/bin/env python1.5

These days, I try to use distutils or setuptools to install scripts, and they can do the right thing to use the correct python to start up anyway...

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mar 14, 2015, at 08:03 AM, Andrew Barnert wrote:

Continuing to use python rather than python2 will work out of the box on nearly every *nix system today, with no exceptions larger than Arch Linux.

The part of the PEP aimed at Python distribution providers should recommend that `python2` be used in preference of just `python` and that the symlink be present. Cheers, -Barry

On Mar 14, 2015, at 08:50 PM, Andrew Barnert wrote:

Maybe in 5-10 years we'll be at a point where you can count on the symlink being there on most *nix systems that have Python 2 pre-installed. Except, of course, that most *nix systems won't have Python 2 pre-installed anymore by then (and those that do will probably overlap quite a bit with the ones that don't have the symlink).

That's the point. By the time Python 2.7 EOLs and (IMHO) we make any change to /usr/bin/python, there will have been plenty of time to make sure anybody running a modern OS version will have the python2 symlink. Given how easy it is to add yourself, I'm not worried about legacy systems or rebels who wouldn't add the link. Cheers, -Barry

On Sun, Mar 15, 2015 at 12:32 AM, Philipp A. <flying-sheep@web.de> wrote:

the only thing worth discussing IMHO is if python 3 should start shipping with all symlinks from past 3.x minor versions to the newest minor version. then we could switch to #!/usr/bin/env python3.4 if it uses e.g. pathlib and would be sure that the script will work with future versions. but then again, users can install pathlib for 3.3…

I would strongly advise NOT doing this. If you say "python3.4", then it should run on Python 3.4, not on Python 3.4-or-anything-later. ChrisA

On Mar 11, 2015, at 06:02 PM, Ryan Gonzalez wrote:

Bad idea? What about when you're running simple scripts you've downloaded and Python 3 is on an external hard drive, not /usr/bin?

I don't get it. Regardless of what a script's shebang says, you can always be explicit about which version you want to use to run the script. You don't *have* to use the shebang's interpreter. Okay, so it's a little more typing - is that really a problem for this use case? Cheers, -Barry

On 11.03.2015 23:45, Andrew Barnert wrote:

On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg <mal@egenix.com> wrote:

...

...

On 11.03.2015 21:03, David Mertz wrote: https://www.python.org/dev/peps/pep-0394/

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems".

So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply.

In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?)

I won't comment on the Mac Python installations. There's a reason why we've been using our own installations since the early days of Max OS X :-) In any case, if those binaries don't exist on the target system, the user will get a nice error message saying e.g.: python2: Command not found. and even better: modern OSes will then point the user to the missing packages on their system to make the script run again. I think that's a lot more informative than SyntaxError: Missing parentheses in call to 'print' :-) Anyway, the above is just what I recommend. I'm not a big fan of doing setuptools like magic all over the place when there's a clean solution right in front of you (ok, one may need to take off those cool Python monkey-patching and meta-class sunglasses occasionally to see it ;-)).

...

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7.

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 12 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 12.03.2015 01:59, Andrew Barnert wrote:

On Mar 11, 2015, at 5:19 PM, M.-A. Lemburg <mal@egenix.com> wrote:

...

...

On 11.03.2015 23:45, Andrew Barnert wrote:

...

On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg <mal@egenix.com> wrote:

...

On 11.03.2015 21:03, David Mertz wrote: https://www.python.org/dev/peps/pep-0394/

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems".

So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply.

In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?)

I won't comment on the Mac Python installations. There's a reason why we've been using our own installations since the early days of Max OS X :-)

In the early days of Mac OS X, Apple shipped an incomplete and broken version of a 4-year-old Python, so that made perfect sense. For half a decade, Apple has been shipping a complete version of the latest 2.x, and with things like PyObjC available out of the box. On top of that, when you tell people to use a separate Python 2.7 installation, they invariably run into problems where they install things for Apple's Python and then try to use them in the other Python or vice-versa, which is compounded by the fact that many people want to use /usr/local/bin as their script directory, but that's the same directory Apple uses, which is one of the ways you get to Ian's problem--whichever ipython was installed last wins--but of course here version numbers don't even help because they're both 2.7. There are hundreds of people on StackOverflow with this problem every year, and surely orders of magnitude more who don't go there for help.

As I said, we've been using our own installations. That doesn't mean the resulting packages won't run with the stock Apple versions, it just means that we're not relying on those Apple versions when compiling our software.

The root problem is that PEP 394 only solves the problem of having two different versions of Python, not two different Pythons of the same version, but you're encouraging people to do the latter. And if PATH doesn't solve it for them (because they're using /usr/local), what does?

This thread is not trying to solve the problem of having multiple Python installations of the same version. The subject problem is migration to Python 3, not running multiple installations of Python 2.7. Let's not get side tracked (I know it's python-ideas and side tracking is fun, but still :-)).

Also, if your program only works on Macs with an extra Python installation, then you don't work on the majority of Unix desktops out there, and it's hard to call that "portable". If you're not interested in working out of the box on most *nix systems, just require Python 3 and all these problems go away. :)

Actually, we don't work much on Macs, but mostly on Linux, FreeBSD and also on Windows, where you have a different set of problems than what's being discussed here, but which can be solved in a similar way by the way of the py launcher: https://www.python.org/dev/peps/pep-0397/ With the right py.ini file, the above approach will work on Windows as well.

...

In any case, if those binaries don't exist on the target system, the user will get a nice error message saying e.g.:

python2: Command not found.

and even better: modern OSes will then point the user to the missing packages on their system to make the script run again.

So nothing other than some of the linux distros counts as a modern OS?

Of course they do. This was just a tangent I couldn't resist, since I think that more modern OSes should provide the same usability features. It's one of the nice Linux distro features I'd love to see in other OSes as well. But this was not the main argument. I think that even the simple python2: Command not found. is better than the standard Python SyntaxError (or other similar Python 3 vs. 2 related error) you get when running a script with the wrong Python version.

...

I think that's a lot more informative than

SyntaxError: Missing parentheses in call to 'print'

:-)

Anyway, the above is just what I recommend.

I'm not a big fan of doing setuptools like magic all over the place when there's a clean solution right in front of you (ok, one may need to take off those cool Python monkey-patching and meta-class sunglasses occasionally to see it ;-)).

...

...

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7.

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 12 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Mar 12, 2015 4:53 PM, "Barry Warsaw" <barry@python.org> wrote:

On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote:

...

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

Mostly, I agree, although I would not recommend /usr/bin/env for system installed scripts. Using /usr/bin/env is fine for development trees, but

a

really bad idea for anything installed in system locations. Modify your suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :)

For (re-)packaged scripts, it seems reasonable to expect package maintainers to use explicit platform-specific paths. Otherwise, for checksums and diff noise, does /usr/bin/env python2 make sense?

I would very much like to see explicit use of /usr/bin/python2 replace /usr/bin/python for scripts.

+1. This will require documented guidelines and many emails to package maintainers. Where should a note about this go in /docs?

Then I think the problem is, what do users expect when they type

$ python foo.py

at their shell prompt?

For the first python binary on $PATH to supersede whatever might be defined in the shebang line of foo.py (thus skipping any shebang path and args).

----- Original Message -----

On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote:

...

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

Mostly, I agree, although I would not recommend /usr/bin/env for system installed scripts. Using /usr/bin/env is fine for development trees, but a really bad idea for anything installed in system locations. Modify your suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :)

+1, the system scripts should use system Python explicitly to keep working when user's environment changes. I'm not sure about Debian/Ubuntu, but we do this in Fedora as much as we can. And while I'm primarily a packager, I also use /usr/bin/env python[2|3] while coding, because it's just more handy during development.

I would very much like to see explicit use of /usr/bin/python2 replace /usr/bin/python for scripts.

+1 (or, better yet, /usr/bin/python3 ;))

Then I think the problem is, what do users expect when they type

$ python foo.py

at their shell prompt?

For more than a year now, I've been trying to push python 3 as default in Fedora. "Default" means that it should be the only Python on live media/cloud images/... and various system applications and system scripts will run on it. I want Fedora to follow PEP 394, so this doesn't mean switching /usr/bin/python to Python 3, but that's one of the biggest sources of confusion. Most users expect /usr/bin/python to point to "the" Python. We're not switching for switching /usr/bin/python (for now) for several reasons: 1) PEP 394 2) System packages still using /usr/bin/python 3) Custom user/sysadmin scripts pointing to /usr/bin/python Point 1 is a recommendation that can change in time, since it's in hands of Python upstream (but then again, it's mostly there because of 3, I think); we'll make point 2 go away in time, because that's something that we can change. Point 3 is the real issue, because (from my experience) there are tons of users/sysadmins that use Python as a shell with nice features and don't really care about what's going on in Python upstream. These scripts are outside of Python upstream and distro control and we can't do anything about them. My thinking is that these won't change - not now, not in two years, not in ten years. The only way to change them is to break them - and then the real question is: do we want to break them or do we want to keep them working forever? (well, at least until we retire Python 2, which would make /usr/bin/python disappear completely) So what can be done: a) PEP 394 changes in few years to recommend pointing /usr/bin/python to Python 3, thus breaking (3). Hopefully this would happen at point where most distros are ready (and willing) to do the change. b) PEP 394 never changes while Python 2 is supported, suggesting switching /usr/bin/python after Python 2 EOL (2020?). c) PEP 394 never changes, not even after Python 2 EOL. d) We remove /usr/bin/python. I've already heard several people suggesting this, but I'm not sure what that should mean for us on distro level. For example, should the system-packaged pip have /usr/bin/pip then? Or should that be removed too? Either way, this would nicely solve (3), since people would just get "python not found" and would have to investigate and learn what's going on, doing an explicit decision about which Python they want to use. On the other hand, I can pretty much guarantee that lots of people would hate this, since they just want to type "python" :) e) Something else? I have to say I prefer a), since I think that we should just move forward at one point and make Python 3 "the Python". My second preference would be d), since that at least doesn't make it look that Python 2 is still "the Python".

Cheers, -Barry

-- Regards, Slavek Kabrda

On Mar 13, 2015, at 05:00 AM, Bohuslav Kabrda wrote:

+1, the system scripts should use system Python explicitly to keep working when user's environment changes. I'm not sure about Debian/Ubuntu, but we do this in Fedora as much as we can.

We do this in Debian/Ubuntu too, although it looks like there are one or two outliers so they probably aren't built with the common Python build tools.

And while I'm primarily a packager, I also use /usr/bin/env python[2|3] while coding, because it's just more handy during development.

Oh, definitely.

For more than a year now, I've been trying to push python 3 as default in Fedora. "Default" means that it should be the only Python on live media/cloud images/... and various system applications and system scripts will run on it.

This is a great goal, and something that we've been working toward in Debian/Ubuntu also. I hope all the Linux distros can do the same thing.

I want Fedora to follow PEP 394, so this doesn't mean switching /usr/bin/python to Python 3, but that's one of the biggest sources of confusion. Most users expect /usr/bin/python to point to "the" Python.

This is what we need to tease out I think. For the most part, users don't care what the shebang line of their scripts is, or even if they have one. They *do* care about what interpreter fires up when they type `python` or any of the variants at their shell, as well as...

3) Custom user/sysadmin scripts pointing to /usr/bin/python

[...]

Point 3 is the real issue, because (from my experience) there are tons of users/sysadmins that use Python as a shell with nice features and don't really care about what's going on in Python upstream. These scripts are outside of Python upstream and distro control and we can't do anything about them. My thinking is that these won't change - not now, not in two years, not in ten years. The only way to change them is to break them - and then the real question is: do we want to break them or do we want to keep them working forever? (well, at least until we retire Python 2, which would make /usr/bin/python disappear completely)

That is indeed the question. I think if we break them, we're just pissing people off for a not compelling enough reason (notice I didn't say "good" reason :). It think the first step is to strongly encourage people to start using /usr/bin/python2 for their Python 2 scripts, and to strongly encourage all distributions of Python to be including a 'python2' symlink. Of course, for anyone with even a rudimentary understanding of *nix, it's easy enough to create that symlink if it's missing, but we might even go a step further and include a 'fix_py2_usr_bin' script that people can run which wouldn't do anything more than ensure /usr/bin/python2 exists. We've committed to bug fix releases for 2.7 until 2020. To me, that's the time frame for any change to /usr/bin/python, whether that's complete removal or repointing the symlink. By then, we'll have 5 more years of educating people to be using 'python2' and 'python3' explicitly so the impact of resolving the ambiguity around 'python' should be lessened. Cheers, -Barry

On Wed, Mar 11, 2015 at 3:10 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

What about using */usr/bin/env*? That wouldn't work; on my Linux system, */usr/bin/env* doesn't like any command like arguments.

Not sure what planet your Linux system is from, but I use #!/usr/bin/env python in all my scripts. It likes the argument ("python" in this case) just fine... Skip

Sorry if this is a double post-didn't reply to all Checking for default version is trivial unless I am missing something .import sys .if sys.version_info[0] < 3: . ## /usr/bin/python2 via subprocess.call or whatever .else: . ##/usr/bin/python3 -- Only when you actually get to the state where there is neither delusion nor enlightenment are you finally comfortable...Foyan

Hi! On Wed, Mar 11, 2015 at 07:58:55PM -0500, Skip Montanaro <skip.montanaro@gmail.com> wrote:

On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" <rymg19@gmail.com> wrote:

...

No; this won't work:

#!/usr/bin/env python -x -3

As Oleg said, anything in the hashbang can only be passed one argument.

It's not the kernel that invokes that; it's the shell.

Why do you think it's the shell? It's exec() system call.

Skip

Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, Mar 12, 2015, at 06:29, Andrew Barnert wrote:

The earliest versions of Unix shbang processing were done in exec, but that turned out to lead to a variety of problems, so most stopped, and after a few decades, POSIX finally (in 2008) forbade it: exec must use the standard shell processor for anything that's not a recognized, loadable executable format.

Show me where POSIX defines "recognized, loadable executable format" to exclude #! as a magic number with its usual behavior. What the standard requires is that execvp and execlp treat what would be a failure of the other members of the exec family of functions (i.e. execve, execle, execv, execl), with [ENOEXEC] by falling back to sh. However, shebang does not cause these other functions to fail with [ENOEXEC], and therefore does not fall under this clause. It does not forbid any processing from being done "in exec" (i.e. in execve).

On Mar 12, 2015, at 8:51 AM, random832@fastmail.us wrote:

Show me where POSIX defines "recognized, loadable executable format" to exclude #! as a magic number with its usual behavior.

I already quoted the standard earlier in the thread, and I already explained how something like linux's pluggable binfmt provides a way around it that's still at least according to the letter of the standard. I don't see much point in repeating what I already wrote. But notice that in the RATIONALE section of the 2001, 2004, 2008, and 2013 versions of the standard (e.g., http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html) effectively describes that as legacy behavior in contrast to the behavior that's "now required by IEEE Std 1003.1-2001":

Another way that some historical implementations handle shell scripts is by recognizing the first two bytes of the file as the character string "#!" and using the remainder of the first line of the file as the name of the command interpreter to execute.

At any rate, this is getting way off track. The point here is that if you're looking for a standard on what #! does everywhere, you're not going to find one. The working group on executable scripts for POSIX-2001 eventually decided to add nothing more than the statement that scripts that start with #! have behavior undefined by the standard, and the recommendation apps that want to use #! should include an installer that sed's the script based on what's on the PATH at install time. Which makes the standard useless if you want to write portable shbang scripts. But practicality beats purity. If all of the *nix platforms that Python currently supports do shbang processing, all have env in /usr/bin (also specifically not required by POSIX, which explicitly says that path locations are not determined by the standard), and none of them swallow all args (as early BSD did), but some drop all args after the first, some pass all, and some pass a single arg with spaces, then that's what you have to work with. You can use "#!/usr/bin/env python", but you can't use "#!/usr/bin/env python -m ipython". Of course if you're building a Mac installer or packages for a particular linux distro or the like, you don't have to worry about portability at all, and can do what makes sense for your platform. (This also means not having to worry about other distros that provide a Python 3.x as /usr/bin/python, etc.)

On Mar 12, 2015, at 1:49 PM, random832@fastmail.us wrote:

Though, speaking of the posix shell script exec issue, and to wrench things back on topic... what about allowing python scripts to start with the line "exec python [options...] $0", a line which will be ignored by the python interpreter?

I vaguely remember either perl or Tcl considering something equivalent long ago as a way of dealing with BSD4.x versions that didn't pass any arguments from the shbang line (before that became irrelevant but env became a de facto practical standard and ate up the first arg, putting us right back in the equivalent place), so it might be worth searching their archives to see what came of that. (Although given the languages we're talking about, I wouldn't be surprised if the answer was "you can already do it in one line in these six different hacky ways, so we don't need to add a special way to do it", which won't apply to Python...) However, I'm not sure this solves the problem the thread is trying to solve. Yes, it gives you a portable way to let 3.5+ scripts pass extra arguments to the interpreter, which might be useful for other reasons (and if so, I think a new thread laying out the rationale would be handy). But the problem people were trying to solve here is that `#!/usr/bin/env python2 -m spam` isn't portable (and neither is `#!/usr/local/bin/spam` where spam is a Python script), and you can't retroactively change 2.x.

On Thu, Mar 12, 2015, at 17:33, Paul Moore wrote:

On 12 March 2015 at 20:49, <random832@fastmail.us> wrote:

...

Though, speaking of the posix shell script exec issue, and to wrench things back on topic... what about allowing python scripts to start with the line "exec python [options...] $0", a line which will be ignored by the python interpreter?

Can't you use the -x option (skip first line of source) for that? Paul

I'd guess the "DOS specific hack" it is intended for is, in fact, precisely "run the python interpreter using the native scripting language in the first line". So... I guess you could. Might be worth cleaning up the line number issue and replacing the env hack with it going forward, given, you know, POSIX compliance.

On Thu, Mar 12, 2015 at 8:21 AM, Skip Montanaro <skip.montanaro@gmail.com> wrote:

... I thought I might have been mistaken or that the field had changed in the meantime.

Which, I see by reading the rest of the thread, it has... S

On Mar 13, 2015, at 09:57 PM, Nick Coghlan wrote:

Right, this most recently came up in the context of Fedora's plans to transition to only having Python 3 in the base install (https://fedoraproject.org/wiki/Changes/Python_3_as_Default). At the moment the expected consequence is that there will come a Fedora version (tentatively Fedora 23) where, out of the box, "/usr/bin/python" and "/usr/bin/env python" will just stop working.

Packages that are part of the distro aren't a problem - those are all being dealt with as the transition progresses, and any which specifically need Python 2 will depend on it accordingly.

The problem arises with custom user packages and scripts, and third party packages and scripts.

Sure, agreed so far. But you're not going to be removing Python 2 from the Fedora archive right? So any user with third party packages that require Python 2 could just install it and their stuff would continue to work. At least that's the plan on Debian/Ubuntu. I'd like to be at the point[*] where the default installs would not contain Python 2, but a simple `apt-get` would install it from the archives for anybody who needs it. That seems like a safer option than changing the symlink. Cheers, -Barry [*] I'm aiming for Debian Stretch and Ubuntu 16.04 Xylophonic X-ray fish LTS.

On Wed, Mar 11, 2015 at 03:26:07PM -0500, Skip Montanaro <skip.montanaro@gmail.com> wrote:

On Wed, Mar 11, 2015 at 3:22 PM, Oleg Broytman <phd@phdru.name> wrote:

...

That wouldn't work. Shebang line accepts one and only one parameter. There are operating systems that allow more but they are exceptions.

If I understood correctly, /usr/bin/python would be started by the kernel, and it would be passed the -x flag.

But it (the kernel) doesn't: $ cat test2 #! /usr/local/bin/python -x -3 print "Ok" $ ./test2 Unknown option: - usage: /usr/local/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ... Try `python -h' for more information. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Mar 11, 2015, at 4:42 PM, random832@fastmail.us wrote:

...

On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: Traditionally the OS only passes the first option to the interpreter so anything after that could be used to pass 3.x specific options.

I think some OSes pass everything after the interpreter as one long string rather than ignoring extra arguments.

And some pass all the arguments separately. Try "#! /usr/bin/env python -O -i" on different platforms; you'll find all three behaviors.

Is there any standard for this?

Nope. POSIX allows for compliant sh implementations to do shbang processing by saying that 'If the first line of a file of shell commands starts with the characters "#!", the results are unspecified". That means there's actually no way to write portable #! scripts according to POSIX. Given that POSIX doesn't require any particular path to env, that's not too surprising; you're already non-portable using /usr/bin/env At least (as of 2008) POSIX does ban doing shbang processing inside the exec functions instead of in sh. (Although even there, Linux's pluggable executable formats provide a way around that...) Somewhere in the POSIX standard, there's a non-normative recommendation that if you want an application to depend on #!, you should include an installer that uses getconf PATH, searches for a suitable interpreter, and used sed to change the shbang to point at the absolute path to that interpreter. So, forget theory; in practice, what's portable? Using exactly one argument works as expected on any system you're likely to find. Which means if you want to use /usr/bin/env instead of using an installer, you don't get any arguments.

On Mar 12, 2015, at 1:24 AM, Oleg Broytman <phd@phdru.name> wrote:

Hi!

...

On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832@fastmail.us wrote:

...

On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: Traditionally the OS only passes the first option to the interpreter so anything after that could be used to pass 3.x specific options.

I think some OSes pass everything after the interpreter as one long string rather than ignoring extra arguments. Is there any standard for this?

There *are* standards ;-) See http://www.in-ulm.de/~mascheck/various/shebang/#splitting and the table below.

A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard.

Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mar 12, 2015, at 3:23 AM, Oleg Broytman <phd@phdru.name> wrote:

...

On Thu, Mar 12, 2015 at 03:17:47AM -0700, Andrew Barnert <abarnert@yahoo.com> wrote: On Mar 12, 2015, at 1:24 AM, Oleg Broytman <phd@phdru.name> wrote:

...

...

...

On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832@fastmail.us wrote: On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: Traditionally the OS only passes the first option to the interpreter so anything after that could be used to pass 3.x specific options.

I think some OSes pass everything after the interpreter as one long string rather than ignoring extra arguments. Is there any standard for this?

There *are* standards ;-) See http://www.in-ulm.de/~mascheck/various/shebang/#splitting and the table below.

A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard.

Isn't that how all standards are implemented in real life?! ;-)

Sure, there are cases where there is a standard but nobody follows it, but that's not the case here: there is no standard to follow in the first place. The relevant standard document (POSIX 2008 or 2013, for most *nixes) explicitly says the behavior is not defined by the standard. (And practically, that makes even more of a difference than it should, as various C libs have at times _intentionally_ done things differently when the standard leaves it open, for good and bad reasons ranging from "so people don't rely on 'it works on Linux and FreeBSD, so it must be standard and portable, so I won't look it up'" to "screw those BSD people for implying something is portable without checking with us".)

-- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mar 12, 2015, at 6:55 AM, Wes Turner <wes.turner@gmail.com> wrote:

On Mar 12, 2015 4:25 AM, "Antoine Pitrou" <solipsis@pitrou.net> wrote:

...

On Wed, 11 Mar 2015 13:43:25 -0600 Neil Schemenauer <nas-python@arctrix.com> wrote:

...

This has been brought up elsewhere, I think this is a better forum to discuss it. Changing /usr/bin/python directly to python3.x is bad idea, in my option. It's going to cause users trouble and the benefit to Python 3.x users is not worth it. Instead, let's try to find a smooth migration path.

As a data point, when you install Python 3 using Conda, the executable is already called "python":

$ python

Brew and pyenv also rely upon $PATH to determine which executable named 'python[maj.min]' to run a script marked as executable with.

+1 for the /usr/bin/env python[maj[.min]] approach is likely most portable (because it uses $PATH for what it is for)

* OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!)) * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent)

2.7.6 on both my base 10.10 box and my 10.10.3 prerelease. (I think it's no longer violating NDA to answer questions about prereleases now that they're public betas rather than DRs... If I'm wrong, pretend I didn't say anything and just consider that they've only changed the Python version in a dot release once in the history of OS X, and that was before they got semi-serious about Python...)

* I think it would be reasonable to request an update that adds a '/usr/bin/python2' symlink, so that paths can be '/usr/bin/env python[maj[.min]'.

Is there someone who can make that request who Apple will listen to more carefully than just a random user's radar? Or should we all just go file radars and link them on OpenRadar for tracking? Also, is there any change of getting it added to 10.9 and 10.10 bug fix releases? Anyway, this would get us a step closer, but as long as there are millions more OS X 10.8, RHEL5, etc. boxes out there (that have python but not python2) than Arch boxes (where python means 3.x); 2.x-only app's still can't portably follow what PEP 394 recommends.

* They seem more focused on (!OpenStack) Swift lately.

It's not as if Python was ever their top priority in the first place. But they've clearly got at least someone working on Python integration given that they included a newer Python 2.7 (and things like newer PyObjC, newer and better-optimized NumPy, PyOpenSSL linked to the same dylib as Python itself, etc.) in each new major OS release, so it's not like there's nobody who could add the symlink...

On Mar 12, 2015 5:00 PM, "Andrew Barnert" <abarnert@yahoo.com> wrote:

On Mar 12, 2015, at 6:55 AM, Wes Turner <wes.turner@gmail.com> wrote:

...

On Mar 12, 2015 4:25 AM, "Antoine Pitrou" <solipsis@pitrou.net> wrote:

...

On Wed, 11 Mar 2015 13:43:25 -0600 Neil Schemenauer <nas-python@arctrix.com> wrote:

...

This has been brought up elsewhere, I think this is a better forum to discuss it. Changing /usr/bin/python directly to python3.x is bad idea, in my option. It's going to cause users trouble and the benefit to Python 3.x users is not worth it. Instead, let's try to find a smooth migration path.

As a data point, when you install Python 3 using Conda, the executable is already called "python":

$ python

Brew and pyenv also rely upon $PATH to determine which executable named

...

+1 for the /usr/bin/env python[maj[.min]] approach is likely most

...

* OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!)) * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent) * I think it would be reasonable to request an update that adds a

'/usr/bin/python2' symlink, so that paths can be '/usr/bin/env

'python[maj.min]' to run a script marked as executable with. portable (because it uses $PATH for what it is for) python[maj[.min]'.

http://openradar.appspot.com/radar?id=5833516721897472

If you want to file your own bug report with Apple, you can mention that

it's a dup of radar 20143311. I sent an email to product-security AT apple.com ( https://www.apple.com/support/security/) with a link to this thread; because TLS.