[Distutils] Shebang lines, /usr/bin/python, and PEP394

[Alex Burke]

On 26 July 2013 13:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 26 July 2013 21:31, Alex Burke <alexjeffburke at gmail.com> wrote:
>> On 26 July 2013 00:37, Philip Jenvey <pjenvey at underboss.org> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA512
>>>
>>>
>>> On Jul 25, 2013, at 9:04 AM, Toshio Kuratomi wrote:
>>>
>>>> Over on python-dev we're talking about Linux Distributions switching from
>>>> python2 to python3, what steps they need to take and in what order.  One of
>>>> the things that's come up [1]_ is that a very early step in the process is making
>>>> sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in
>>>> PEP394 [2]_.  Faced with the prospect of patching a whole bunch of scripts
>>>> in the distribution, I'm wondering what distutils, distlib, setuptools, etc
>>>> do with shebang lines.
>>>> * Do they rewrite shebang lines?
>>>
>>> distutils, distlib and setuptools all do.
>>
>> Hi,
>>
>> It was interesting that discussion came up on python-dev but I admit
>> to being surprised by the suggestion the shebang lines may need to be
>> rewritten in end user code.
>>
>> This may be a callous over-simplification but if #!python is rewritten
>> by the python packaging infrastructure, would it not be changed for
>> python2/python3 as appropriate at installation time? Thus a python 2
>> package (whatever it is named) would be generated by calling a python2
>> executable + setuptools while the same is true for v3 except using
>> python3. The result is then packaged by rpm/dpkg.
>>
>> Keen to understand why it can't work this way if that's the case.
>
> It actually occurs to me that the following example (showing how
> symlinks affect "sys.executable") illustrates both the problem and the
> solution for cases where users are relying on generated script
> wrappers or the shebang line rewriting in wheel:
>
>     $ ln -s /usr/bin/python foo
>     $ ./foo -c "import sys; print sys.executable"
>     /home/ncoghlan/foo
>
> So long as the distro build systems are updated to invoke setup.py
> through an appropriately versioned symlink (rather than through
> /usr/bin/python), setuptools et al should all automatically do the
> right thing when generating script wrappers.

That's pretty much exactly the mechanism I had in mind.

> Not everybody uses generated script wrappers, though - if there is a
> hardcoded "/usr/bin/env python" or "/usr/bin/python" in a shebang
> line, the Python build tools won't touch it. There's also a whole lot
> of software that isn't packaged at all - it's sitting around in user
> and admin home directories, or maybe in a shared directory on a file
> server or even in a private source control repo.
>
> Published software is actually the vanishingly small tip of a very
> large iceberg, especially for languages like Python that handle
> scripting use cases fairly well and are thus popular for personal
> automation tasks amongst developers and system administrators.

Hmm, that's a very good point. I guess I'd been considering packaged
software or at least things that installed through a distribution's package
manager.

That being said, if this is a sound approach I reckon the advice issued
to packagers could be always use the shebang line if updating software
that gets packages, and otherwise any other arbitrary 'env python' just
defers to the top level 'python' symlink which perhaps is best decided
by the system administrator themselves. Re the python-dev discussion
does this actually act in favour of a python2/python3 one of which is
chosen to be active?

> Cheers,
> Nick.

Btw one more thought sprang to mind - may be entirely unfeasible but the
un-packaged software case made me think it could be interesting to
have a pip 'installscript' or python -m setuptools.installscipt (the second
example for illustration purposes only!) that you could point at an
arbitrary script; it places it in the bin directory and does a shebang swap.

Thanks, Alex.