PEP 379 Python launcher for Windows - behaviour for #!/usr/bin/env python line is wrong
While reviewing the behaviour of Vinay's "distil" installer tool (see distutils-sig for details, but it's not relevant here) I have found what I think is a flaw in the behaviour of the py.exe launcher for Windows. To recap for people unfamiliar with the launcher, it emulates #! line interpretation on Windows, interpreting commonly used forms from Unix and launching the appropriate installed Python interpreter (finding its location from the registry, as python.exe may not be on PATH). The problem is with the interpretation of #!/usr/bin/env python. The launcher treats this the same as #!/usr/bin/python, launching the "default" Python. But that is *not* what the equivalent line does on Unix, where it launches the *currently active* Python (a crucial difference when there is an active virtualenv). The result is that a script written to run with the active Python works on Unix as expected, but can use an unexpected version of Python on Windows. This is particularly unpleasant when the program in question is an (un)installer like distil! I would propose that the behaviour of the launcher on Windows should be changed when it encounters specifically the hashbang line #!/usr/bin/env python. In that case, it should search PATH for a copy of python.exe, and if it finds one, use that. If there is no python.exe on PATH, it should fall back to the same version of Python as would have been used if the line were #!/usr/bin/python. This will mean that scripts written with #!/usr/bin/env python will behave the same on Unix and Windows in the presence of activated virtualenvs. Would people be happy with this change? If so I will open an issue on bugs.python.org. I can look at producing a patch, as well. Paul
On Fri, May 3, 2013 at 3:23 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I would propose that the behaviour of the launcher on Windows should be changed when it encounters specifically the hashbang line #!/usr/bin/env python. In that case, it should search PATH for a copy of python.exe, and if it finds one, use that. If there is no python.exe on PATH, it should fall back to the same version of Python as would have been used if the line were #!/usr/bin/python.
This will mean that scripts written with #!/usr/bin/env python will behave the same on Unix and Windows in the presence of activated virtualenvs.
Would people be happy with this change? If so I will open an issue on bugs.python.org. I can look at producing a patch, as well.
Sounds reasonable to me.
On Sat, May 4, 2013 at 12:18 PM, Brian Curtin <brian@python.org> wrote:
On Fri, May 3, 2013 at 3:23 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I would propose that the behaviour of the launcher on Windows should be changed when it encounters specifically the hashbang line #!/usr/bin/env python. In that case, it should search PATH for a copy of python.exe, and if it finds one, use that. If there is no python.exe on PATH, it should fall back to the same version of Python as would have been used if the line were #!/usr/bin/python.
This will mean that scripts written with #!/usr/bin/env python will behave the same on Unix and Windows in the presence of activated virtualenvs.
Would people be happy with this change? If so I will open an issue on bugs.python.org. I can look at producing a patch, as well.
Sounds reasonable to me.
Also to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Paul Moore <p.f.moore <at> gmail.com> writes:
This will mean that scripts written with #!/usr/bin/env python will behave the same on Unix and Windows in the presence of activated virtualenvs.
Overall I think it's the right result. There's one other compatibility clarification: at the moment, as allowed by the PEP, you can have launcher flags in a line that starts with #!/usr/bin/env python, for example #!/usr/bin/env python3.2-32 -u In such a case the launcher would use the 3.2-32 suffix to indicate that 32-bit Python 3.2 is wanted, and pass the -u to the launched executable. I assume that this behaviour should continue, and the Posix-compatible behaviour being proposed should only apply for lines that contain "#!/usr/bin/env python" followed by whitespace. Also, since we're making a backwards incompatible change, do people feel that it needs to be switched on only in the presence of e.g. an environment variable such as PYLAUNCH_SEARCHPATH, or should we just change the default behaviour now and risk breaking user scripts which may rely on the current behaviour? Regards, Vinay Sajip
On 4 May 2013 15:20, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
This will mean that scripts written with #!/usr/bin/env python will behave the same on Unix and Windows in the presence of activated virtualenvs.
Overall I think it's the right result. There's one other compatibility clarification: at the moment, as allowed by the PEP, you can have launcher flags in a line that starts with #!/usr/bin/env python, for example
#!/usr/bin/env python3.2-32 -u
In such a case the launcher would use the 3.2-32 suffix to indicate that 32-bit Python 3.2 is wanted, and pass the -u to the launched executable. I assume that this behaviour should continue, and the Posix-compatible behaviour being proposed should only apply for lines that contain
"#!/usr/bin/env python"
followed by whitespace.
That sounds reasonable - I've never used the ability to add flags, but I agree that as there's no equivalent in POSIX, there's no reason to change the current behaviour in that case.
Also, since we're making a backwards incompatible change, do people feel that it needs to be switched on only in the presence of e.g. an environment variable such as PYLAUNCH_SEARCHPATH, or should we just change the default behaviour now and risk breaking user scripts which may rely on the current behaviour?
Personally, I'd say make it unconditional - the current behaviour is unlikely to ever be what people actually *want*. But if the consensus is to make it conditional, could we have a flag in py.ini rather than an environment variable? That would be more consistent with normal Windows practice. Paul. PS Vinay - from this post, I assume you're already looking at this code? I was considering trying to put together a patch, but I don't want to duplicate effort if you're working on it.
Paul Moore <p.f.moore <at> gmail.com> writes:
<div>PS Vinay - from this post, I assume you're already looking at this code? I was considering trying to put together a patch, but I don't want to duplicate effort if you're working on it.</div>
I've taken a quick look at it, but I probably won't be able to make any changes until the near the end of the coming week. Feel free to have a go; the place to make changes will be near the call is_virt = parse_shebang(...) ... if (!is_virt) { ... } else { /* In here is where the new logic will probably go. */ } Also, the #define SEARCH_PATH needs to be uncommented to include the find_on_path function. It also enables searching the path for customised commands. Regards, Vinay
On 4 May 2013 16:42, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I've taken a quick look at it, but I probably won't be able to make any changes until the near the end of the coming week. Feel free to have a go;
OK, I have a patch against the standalone pylauncher repo at https://bitbucket.org/pmoore/pylauncher. I'm not sure what the best approach is - I didn't want to patch the python core version directly (a) because I wouldn't be able to test it easily, and (b) because I'd want a standalone version anyway until 3.4 comes out. BTW, the tests for pylauncher fail for me on the unpatched version, so all I can say is that the patched version fails the same way and my manual tests worked as expected... I can rework it against cpython if needed. Paul.
On 5 May 2013 18:10, Paul Moore <p.f.moore@gmail.com> wrote:
On 4 May 2013 16:42, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I've taken a quick look at it, but I probably won't be able to make any changes until the near the end of the coming week. Feel free to have a go;
OK, I have a patch against the standalone pylauncher repo at https://bitbucket.org/pmoore/pylauncher. I'm not sure what the best approach is - I didn't want to patch the python core version directly (a) because I wouldn't be able to test it easily, and (b) because I'd want a standalone version anyway until 3.4 comes out.
Vinay, Did you get a chance to have a look at this? I didn't manage to create a pull request against your copy of pylauncher as my repo is a fork of the pypa one - I'm not sure if that's a limitation of bitbucket or if I just don't know how to do it... I've created a pull request against the pypa version in case that's of use... Paul
From: Paul Moore <p.f.moore@gmail.com>
Did you get a chance to have a look at this? I didn't manage to create a pull request against your copy of pylauncher as my repo is a fork of the pypa one - I'm not sure if that's a limitation of bitbucket or if I just don't know how to do it... I've created a pull request against the pypa version in case that's of use...
Hi Paul, Sorry I haven't had a chance yet - real life is very busy at the moment. A pull request against the pypa version is fine, and I will get to it soon - thanks for your patience. Regards, Vinay Sajip
participants (4)
-
Brian Curtin -
Nick Coghlan -
Paul Moore -
Vinay Sajip