Shebang lines, /usr/bin/python, and PEP394
Over on python-dev we're talking about Linux Distributions switching from python2 to python3, what steps they need to take and in what order. One of the things that's come up [1]_ is that a very early step in the process is making sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in PEP394 [2]_. Faced with the prospect of patching a whole bunch of scripts in the distribution, I'm wondering what distutils, distlib, setuptools, etc do with shebang lines. * Do they rewrite shebang lines? * If so, do they use #!/usr/bin/python2 or do they use #!/usr/bin/python ? * If the latter, is there hope that we could change that to match with PEP-394's recommendations? (setuptools seems to be moving relatively quickly these days so that seems reasonably easy.... distutils is tied to the release schedule of core python-2.7.x although if the change is accepted into the CPython tree we might consider backporting it to the current distribution package early. .. [1]_: http://mail.python.org/pipermail/python-dev/2013-July/127565.html .. [2]_: http://www.python.org/dev/peps/pep-0394/#recommendation Thanks, Toshio
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Jul 25, 2013, at 9:04 AM, Toshio Kuratomi wrote:
Over on python-dev we're talking about Linux Distributions switching from python2 to python3, what steps they need to take and in what order. One of the things that's come up [1]_ is that a very early step in the process is making sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in PEP394 [2]_. Faced with the prospect of patching a whole bunch of scripts in the distribution, I'm wondering what distutils, distlib, setuptools, etc do with shebang lines. * Do they rewrite shebang lines?
distutils, distlib and setuptools all do.
* If so, do they use #!/usr/bin/python2 or do they use #!/usr/bin/python ?
I believe they are actually all using sys.executable
* If the latter, is there hope that we could change that to match with PEP-394's recommendations? (setuptools seems to be moving relatively quickly these days so that seems reasonably easy.... distutils is tied to the release schedule of core python-2.7.x although if the change is accepted into the CPython tree we might consider backporting it to the current distribution package early.
What's the value of sys.executable in the brave new world of distributions that follow PEP 394? - -- Philip Jenvey -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJR8ai9AAoJEJ82HVHb51bRPsUH/0eH03riVb1EtruYxqQ0sFxZ F0XhVPSdSfMgU63e1aXTeLs1KT+NwHjH5w25Zt024/Xizv/HhDFr8ElfsPjCZwW3 fUBUuSOcq9zuLsDuAvsRXPGuv5jRSZf8DiRExDFQjV1GZSuZ46ExOf3D/TJmbcOf EibAzPzDCOZdgGTw4NcS2yzE9N/woyIjSyvJPXG8IyyFEFw9V4ZfpZL5uiKoKt0p acbC8tzh0w79atsHN1tCalYhcXzIgAsAWBpXj18V4RKxjtZW9i1k5KQKmkKn2FLF Vr5ELNjaoLFJ77RuBZWGN6GOv5rPh1C9PFzFRURrrSk4CWEma636GuktbpA3Kik= =zzg5 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Jul 25, 2013, at 03:37 PM, Philip Jenvey wrote:
What's the value of sys.executable in the brave new world of distributions that follow PEP 394?
One use case is locating other scripts/executables that might get installed next to sys.executable in a virtualenv. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) iQIcBAEBCAAGBQJR8cZfAAoJEBJutWOnSwa/lnMQALE6VLkSNElQfJnIgBraS/PR /C7m2kCNbeBdHfTXjRJYeHoecFPO6z5mSurO2ikbFvwvxmWi1MjxX6/gKR6jaNJ6 PJcyoO1KaJJ9CKP0hlAcuLYGyj86Nhmw1cK20yeejV+YJzqpFjqXxHyH9GlaGQKc aDKG/xN/SX139V4/KfgaAcXKhbsCkhL6cAugHFGl8qTytdvaOOzRs2/jHsTcPDp3 Dfb2nz69PteQAYSPHi7Na9bAe2+lGy9nz93zfw7T6FLB4ya3trATnbxD1Ojmg2mw lnuvTBg6GiRv/BZgDB6Fl8OTMj6BvyfKKG9Nb6rCCCYyPR8Q8t1TkxODCAkHi6kY omwMOR+Fc4Btm/rtRcY4rXKPY6FXVgnzysOT/z4GcpjWotp2E6ZiSruwsm7okwM3 Pbl9BEAD4WdgotSJDrb17BmcLl48qqenODOmrHKvfIu67iagSAr8D9Qp2B0G1V0B GRtmCVrcvGheMKw+5VAYAmfyVnNQmi1MVY1fNN9pya9Wi38mYiWFxXhD/wIhU6Ih /vs38dt+PSu6AMfomvf7tzOuZKE+JKkS//e7+OlEwwaDl4am5awXh0Ci/6z8cJHg ywH003hYZ4SWzuMHWerkIxydXW3D8GUvRDcpR3OZaoknEKQYkGoMVY0KimKG2ykZ KcGciNDFFXpxaq6+4cc1 =Dy/O -----END PGP SIGNATURE-----
On 26 Jul 2013 10:45, "Barry Warsaw" <barry@python.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Jul 25, 2013, at 03:37 PM, Philip Jenvey wrote:
What's the value of sys.executable in the brave new world of
distributions
that follow PEP 394?
One use case is locating other scripts/executables that might get installed next to sys.executable in a virtualenv.
I believe Philip's question was literal rather than philosophical :) Cheers, Nick.
- -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux)
iQIcBAEBCAAGBQJR8cZfAAoJEBJutWOnSwa/lnMQALE6VLkSNElQfJnIgBraS/PR /C7m2kCNbeBdHfTXjRJYeHoecFPO6z5mSurO2ikbFvwvxmWi1MjxX6/gKR6jaNJ6 PJcyoO1KaJJ9CKP0hlAcuLYGyj86Nhmw1cK20yeejV+YJzqpFjqXxHyH9GlaGQKc aDKG/xN/SX139V4/KfgaAcXKhbsCkhL6cAugHFGl8qTytdvaOOzRs2/jHsTcPDp3 Dfb2nz69PteQAYSPHi7Na9bAe2+lGy9nz93zfw7T6FLB4ya3trATnbxD1Ojmg2mw lnuvTBg6GiRv/BZgDB6Fl8OTMj6BvyfKKG9Nb6rCCCYyPR8Q8t1TkxODCAkHi6kY omwMOR+Fc4Btm/rtRcY4rXKPY6FXVgnzysOT/z4GcpjWotp2E6ZiSruwsm7okwM3 Pbl9BEAD4WdgotSJDrb17BmcLl48qqenODOmrHKvfIu67iagSAr8D9Qp2B0G1V0B GRtmCVrcvGheMKw+5VAYAmfyVnNQmi1MVY1fNN9pya9Wi38mYiWFxXhD/wIhU6Ih /vs38dt+PSu6AMfomvf7tzOuZKE+JKkS//e7+OlEwwaDl4am5awXh0Ci/6z8cJHg ywH003hYZ4SWzuMHWerkIxydXW3D8GUvRDcpR3OZaoknEKQYkGoMVY0KimKG2ykZ KcGciNDFFXpxaq6+4cc1 =Dy/O -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On 26 July 2013 00:37, Philip Jenvey <pjenvey@underboss.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Jul 25, 2013, at 9:04 AM, Toshio Kuratomi wrote:
Over on python-dev we're talking about Linux Distributions switching from python2 to python3, what steps they need to take and in what order. One of the things that's come up [1]_ is that a very early step in the process is making sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in PEP394 [2]_. Faced with the prospect of patching a whole bunch of scripts in the distribution, I'm wondering what distutils, distlib, setuptools, etc do with shebang lines. * Do they rewrite shebang lines?
distutils, distlib and setuptools all do.
Hi, It was interesting that discussion came up on python-dev but I admit to being surprised by the suggestion the shebang lines may need to be rewritten in end user code. This may be a callous over-simplification but if #!python is rewritten by the python packaging infrastructure, would it not be changed for python2/python3 as appropriate at installation time? Thus a python 2 package (whatever it is named) would be generated by calling a python2 executable + setuptools while the same is true for v3 except using python3. The result is then packaged by rpm/dpkg. Keen to understand why it can't work this way if that's the case. Thanks, Alex J Burke.
On 26 July 2013 21:31, Alex Burke <alexjeffburke@gmail.com> wrote:
On 26 July 2013 00:37, Philip Jenvey <pjenvey@underboss.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Jul 25, 2013, at 9:04 AM, Toshio Kuratomi wrote:
Over on python-dev we're talking about Linux Distributions switching from python2 to python3, what steps they need to take and in what order. One of the things that's come up [1]_ is that a very early step in the process is making sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in PEP394 [2]_. Faced with the prospect of patching a whole bunch of scripts in the distribution, I'm wondering what distutils, distlib, setuptools, etc do with shebang lines. * Do they rewrite shebang lines?
distutils, distlib and setuptools all do.
Hi,
It was interesting that discussion came up on python-dev but I admit to being surprised by the suggestion the shebang lines may need to be rewritten in end user code.
This may be a callous over-simplification but if #!python is rewritten by the python packaging infrastructure, would it not be changed for python2/python3 as appropriate at installation time? Thus a python 2 package (whatever it is named) would be generated by calling a python2 executable + setuptools while the same is true for v3 except using python3. The result is then packaged by rpm/dpkg.
Keen to understand why it can't work this way if that's the case.
It actually occurs to me that the following example (showing how symlinks affect "sys.executable") illustrates both the problem and the solution for cases where users are relying on generated script wrappers or the shebang line rewriting in wheel: $ ln -s /usr/bin/python foo $ ./foo -c "import sys; print sys.executable" /home/ncoghlan/foo So long as the distro build systems are updated to invoke setup.py through an appropriately versioned symlink (rather than through /usr/bin/python), setuptools et al should all automatically do the right thing when generating script wrappers. Not everybody uses generated script wrappers, though - if there is a hardcoded "/usr/bin/env python" or "/usr/bin/python" in a shebang line, the Python build tools won't touch it. There's also a whole lot of software that isn't packaged at all - it's sitting around in user and admin home directories, or maybe in a shared directory on a file server or even in a private source control repo. Published software is actually the vanishingly small tip of a very large iceberg, especially for languages like Python that handle scripting use cases fairly well and are thus popular for personal automation tasks amongst developers and system administrators. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 26, 2013, at 09:58 PM, Nick Coghlan wrote:
Not everybody uses generated script wrappers, though - if there is a hardcoded "/usr/bin/env python" or "/usr/bin/python" in a shebang line, the Python build tools won't touch it. There's also a whole lot of software that isn't packaged at all - it's sitting around in user and admin home directories, or maybe in a shared directory on a file server or even in a private source control repo.
I actually think installing a script via setuptools should rewrite shebangs even in the case of /usr/bin/env. I love `#!/usr/bin/env python` *for development* but I really think its a bad thing to have for installed scripts. Certainly, for distro installed scripts, it's (usually) terrible. I think virtualenv installs are generally in the same boat though - if you're installing a script into a virtualenv, it's better to rewrite the shebang to use the executable that was used to install it. -Barry
On 26 July 2013 20:32, Barry Warsaw <barry@python.org> wrote:
I love `#!/usr/bin/env python` *for development* but I really think its a bad thing to have for installed scripts. Certainly, for distro installed scripts, it's (usually) terrible. I think virtualenv installs are generally in the same boat though - if you're installing a script into a virtualenv, it's better to rewrite the shebang to use the executable that was used to install it.
There are cases where it's useful and appropriate - the best example "in the wild" is distil, which uses #!/usr/bin/env python to allow it to run with "the current Python" specifically, so that it can be used to install packages whatever Python you're using and without needing multiple copies of the script installed. I've written similar types of scripts which benefit in the same way from a /user/bin/env shebang line. In fact, support for /usr/bin/env was added to the Windows launcher just recently to provide precisely this functionality for scripts on Windows. Paul
On Jul 26, 2013, at 08:38 PM, Paul Moore wrote:
There are cases where it's useful and appropriate
Sure, I don't disagree. Just that I think the general rule should be: * Use /usr/bin/env in your source tree * Use /usr/bin/$python when installed I think those rules cover the majority of cases. Any automatic shebang rewriting must be selectable. I'd argue for default-rewrite with an option to disable. -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/26/2013 03:32 PM, Barry Warsaw wrote:
If you're installing a script into a virtualenv, it's better to rewrite the shebang to use the executable that was used to install it.
Exactly -- the script likely won't run at all outside the environment where its dependencies are installed. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHy4E0ACgkQ+gerLs4ltQ57XgCg1JsuCk2zxOpJyWNoHv1V/0h/ Y1EAoND2clmyfdHjqVY/3p+PafWXM0Lp =uzfB -----END PGP SIGNATURE-----
On 26 July 2013 13:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 26 July 2013 21:31, Alex Burke <alexjeffburke@gmail.com> wrote:
On 26 July 2013 00:37, Philip Jenvey <pjenvey@underboss.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Jul 25, 2013, at 9:04 AM, Toshio Kuratomi wrote:
Over on python-dev we're talking about Linux Distributions switching from python2 to python3, what steps they need to take and in what order. One of the things that's come up [1]_ is that a very early step in the process is making sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in PEP394 [2]_. Faced with the prospect of patching a whole bunch of scripts in the distribution, I'm wondering what distutils, distlib, setuptools, etc do with shebang lines. * Do they rewrite shebang lines?
distutils, distlib and setuptools all do.
Hi,
It was interesting that discussion came up on python-dev but I admit to being surprised by the suggestion the shebang lines may need to be rewritten in end user code.
This may be a callous over-simplification but if #!python is rewritten by the python packaging infrastructure, would it not be changed for python2/python3 as appropriate at installation time? Thus a python 2 package (whatever it is named) would be generated by calling a python2 executable + setuptools while the same is true for v3 except using python3. The result is then packaged by rpm/dpkg.
Keen to understand why it can't work this way if that's the case.
It actually occurs to me that the following example (showing how symlinks affect "sys.executable") illustrates both the problem and the solution for cases where users are relying on generated script wrappers or the shebang line rewriting in wheel:
$ ln -s /usr/bin/python foo $ ./foo -c "import sys; print sys.executable" /home/ncoghlan/foo
So long as the distro build systems are updated to invoke setup.py through an appropriately versioned symlink (rather than through /usr/bin/python), setuptools et al should all automatically do the right thing when generating script wrappers.
That's pretty much exactly the mechanism I had in mind.
Not everybody uses generated script wrappers, though - if there is a hardcoded "/usr/bin/env python" or "/usr/bin/python" in a shebang line, the Python build tools won't touch it. There's also a whole lot of software that isn't packaged at all - it's sitting around in user and admin home directories, or maybe in a shared directory on a file server or even in a private source control repo.
Published software is actually the vanishingly small tip of a very large iceberg, especially for languages like Python that handle scripting use cases fairly well and are thus popular for personal automation tasks amongst developers and system administrators.
Hmm, that's a very good point. I guess I'd been considering packaged software or at least things that installed through a distribution's package manager. That being said, if this is a sound approach I reckon the advice issued to packagers could be always use the shebang line if updating software that gets packages, and otherwise any other arbitrary 'env python' just defers to the top level 'python' symlink which perhaps is best decided by the system administrator themselves. Re the python-dev discussion does this actually act in favour of a python2/python3 one of which is chosen to be active?
Cheers, Nick.
Btw one more thought sprang to mind - may be entirely unfeasible but the un-packaged software case made me think it could be interesting to have a pip 'installscript' or python -m setuptools.installscipt (the second example for illustration purposes only!) that you could point at an arbitrary script; it places it in the bin directory and does a shebang swap. Thanks, Alex.
participants (8)
-
Alex Burke -
Barry Warsaw -
Nick Coghlan -
Paul Moore -
Philip Jenvey -
Toshio Kuratomi -
Tres Seaver -
Vinay Sajip