On Mon, 28 May 2012 19:37:55 +0000 (UTC) Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Antoine Pitrou <solipsis <at> pitrou.net> writes:

...

Regardless of what the executable is or does, its source code must be included somewhere in the Python source tree (and, preferably, there should be a simple procedure to build the binaries).

I understand that. Does it need to be checked in right now?

Not necessarily, but OTOH, it is not really standard procedure to commit half-finished patches. Regards Antoine.

On Mon, 28 May 2012 21:23:50 +0000 (UTC) Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Antoine Pitrou <solipsis <at> pitrou.net> writes:

...

Not necessarily, but OTOH, it is not really standard procedure to commit half-finished patches.

I didn't want to miss the window for the upcoming alpha, and and I'm not sure exactly how things will pan out with respect to PEP 397 and packaging. If people generally feel strongly about this, I can delete the .exe and re-introduce it later if/when appropriate. It might have had a few rough edges, but I wouldn't have characterised the patch as "half-finished" - that seems a little harsh :-)

Yes, I shouldn't have said that. "Unfinished" is more appropriate. Regards Antoine.

Vinay originally wrote:

PEP 397 (Python launcher for Windows) has not yet been accepted, so there still needs to be some way of natively launching scripts in Windows which is equivalent to /path/to/venv/bin/foo. The way setuptools (and hence Distribute) does this is to shadow each script with an executable: whereas a script foo would be simply placed in /path/to/venv/bin/ on POSIX, on Windows, the files foo.exe and foo-script.py (or foo-script.pyw) are placed in \path\to\venv\Scripts. The user can run \path\to\venv\Scripts\foo just as on POSIX.

The foo.exe file is just a copy of a stock launcher executable which finds its name from the C argv[0], and based in that name (foo in this case), invokes foo-script.py or foo-script.pyw with the appropriate Python interpreter.

I don't understand the relationship between this "stock launcher" and the PEP 397 launcher. They seem to have quite distinct requirements without much overlap. Specifically, I'm not aware that the current PEP 397 implementation could perform the same role as the "stock launcher" - IIUC, it has no special handling of the "-script" suffix or special logic based around its argv[0]. So unless I'm mistaken, even with PEP 397 accepted, either this "stock launcher" is still necessary anyway or the PEP398 launcher would need the addition of new features so it could replace the stock launcher. FWIW, Vinay and I exchanged some private mail recently about how to best integrate the PEP397 launcher with virtualenvs - and while we both agreed it would be nice, we couldn't come up with anything worthwhile. Having the launcher be aware of a virtualenv when invoked via file associations is problematic - for example, Windows Explorer is unlikely to have a virtualenv configured in its environment. Even when considering just command-line usage there are some edge-cases that make things problematic (eg, what if a script in a venv nominates a specific Python version via a shebang line? What if a venv is activated but the user/launcher attempts to execute a script outside the venv? etc.) On 29/05/2012 9:24 AM, Nick Coghlan wrote: ...

It would have been better if the issue of script management on Windows had been raised in PEP 405 itself - I likely would have declared PEP 397 a dependency *before* accepting it (even if that meant the feature missed the alpha 4 deadline and first appeared in beta 1, or potentially even missed 3.3 altogether).

However, I'm not going to withdraw the acceptance of the PEP over this - while I would have made a different decision at the time given the additional information (due to the general preference to treat Windows as a first class deployment target), I think reversing my decision now would make the situation worse rather than better.

That means the important question is what needs to happen before beta 1 at the end of June. As I see it, we have two ways forward:

1. My preferred option: bring PEP 397 up to scratch as a specification for the behaviour of the Python launcher (perhaps with Vinay stepping up as a co-author to help Mark if need be), find a BDFL delegate (MvL? Brian Curtin?) and submit that PEP for acceptance within the next few weeks. The updated PEP 397 should include an explanation of exactly how it will help with the correct implementation of PEP 405 on Windows (this may involve making the launcher pyvenv aware).

As above, it isn't clear to me how the additional complexity and list of caveats in real use make it worthwhile to have the PEP397 launcher pyvenv aware.

2. The fallback option: remove the currently checked in build artifacts from source control and incorporate them into the normal Windows build processes (both the main VS 2010 process, and at least the now-legacy VS 2008 process)

For alpha 4, I suggest going with MvL's suggestion - drop the binaries from Mercurial and accept that this aspect of PEP 405 simply won't work on Windows until the first beta.

Agreed - ISTM that this stock launcher is probably going to need to co-exist with the PEP397 launcher for the long term. Cheers, Mark

On 05/28/2012 04:24 PM, Nick Coghlan wrote:

It would have been better if the issue of script management on Windows had been raised in PEP 405 itself - I likely would have declared PEP 397 a dependency *before* accepting it (even if that meant the feature missed the alpha 4 deadline and first appeared in beta 1, or potentially even missed 3.3 altogether).

However, I'm not going to withdraw the acceptance of the PEP over this - while I would have made a different decision at the time given the additional information (due to the general preference to treat Windows as a first class deployment target), I think reversing my decision now would make the situation worse rather than better.

I think it's unfortunate that this issue (which is http://bugs.python.org/issue12394) has become entangled with PEP 405 at all, since AFAICT it is entirely orthogonal. This is a distutils2/packaging issue regarding how scripts are installed on Windows. It happens to be relevant when trying to install things into a PEP 405 venv on Windows, but it applies to a non-virtual Python installation on Windows every bit as much as it applies to a PEP 405 environment. In an earlier discussion with Vinay I thought we had agreed that it was an orthogonal issue and that this proposed patch for it would be removed from the PEP 405 reference implementation before it was merged to CPython trunk; I think that would have been preferable. This is why there is no mention of the issue in PEP 405 - it doesn't belong there, because it is not related.

That means the important question is what needs to happen before beta 1 at the end of June. As I see it, we have two ways forward:

1. My preferred option: bring PEP 397 up to scratch as a specification for the behaviour of the Python launcher (perhaps with Vinay stepping up as a co-author to help Mark if need be), find a BDFL delegate (MvL? Brian Curtin?) and submit that PEP for acceptance within the next few weeks. The updated PEP 397 should include an explanation of exactly how it will help with the correct implementation of PEP 405 on Windows (this may involve making the launcher pyvenv aware).

2. The fallback option: remove the currently checked in build artifacts from source control and incorporate them into the normal Windows build processes (both the main VS 2010 process, and at least the now-legacy VS 2008 process)

For alpha 4, I suggest going with MvL's suggestion - drop the binaries from Mercurial and accept that this aspect of PEP 405 simply won't work on Windows until the first beta.

Regardless, these sound like the right options moving forward, with the clarification that it is not any "aspect of PEP 405" that will not work until a fix is merged, it is simply an existing limitation of distutils2/packaging on Windows. And that if anything needs to be reverted, temporarily or permanently, it should not be all of the PEP 405 implementation, rather just this packaging fix. Carl

Nick Coghlan <ncoghlan <at> gmail.com> writes:

In that case: Vinay, please revert everything from the pyvenv commit that was actually related to issue #12394 rather than being part of the PEP 405 implementation. As Carl says, it's an unrelated change that needs to be discussed separately.

There's nothing in there related to #12394 - just a pysetup3.exe file, which I had originally overlooked and have now removed. Regards, Vinay Sajip

Am 29.05.2012 01:15, schrieb martin@v.loewis.de:

...

Comments welcome, especially on whether Windows users agree that something like this is needed in the absence of PEP 397 in Python 3.3.

AFAICT, there is no need to check in the binary into revision control. Instead, the Windows build process should create, package, and deploy them, and venv should then just expect that they are there.

So I request that this checkin is reverted, preferably before the alpha release.

I also agree with the fundamental principle that an open source project should never ever include binaries for which it doesn't also provide source code. If you cannot release the sources right now, do not release the binaries either.

Agreed. Vinay, please either let me know when this is rectified (see also Nick's request about reverting #12394 specific parts of the commit), or revert the whole PEP 405 implementation for now, if the time is too short: I don't want to delay the alpha much longer. There is still time until beta after all. Georg

On Mon, 28 May 2012 19:37:55 +0000 (UTC) Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Antoine Pitrou <solipsis <at> pitrou.net> writes:

...

Regardless of what the executable is or does, its source code must be included somewhere in the Python source tree (and, preferably, there should be a simple procedure to build the binaries).

I understand that. Does it need to be checked in right now?

Not necessarily, but OTOH, it is not really standard procedure to commit half-finished patches. Regards Antoine.

On Mon, 28 May 2012 21:23:50 +0000 (UTC) Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Antoine Pitrou <solipsis <at> pitrou.net> writes:

...

Not necessarily, but OTOH, it is not really standard procedure to commit half-finished patches.

I didn't want to miss the window for the upcoming alpha, and and I'm not sure exactly how things will pan out with respect to PEP 397 and packaging. If people generally feel strongly about this, I can delete the .exe and re-introduce it later if/when appropriate. It might have had a few rough edges, but I wouldn't have characterised the patch as "half-finished" - that seems a little harsh :-)

Yes, I shouldn't have said that. "Unfinished" is more appropriate. Regards Antoine.

Vinay originally wrote:

PEP 397 (Python launcher for Windows) has not yet been accepted, so there still needs to be some way of natively launching scripts in Windows which is equivalent to /path/to/venv/bin/foo. The way setuptools (and hence Distribute) does this is to shadow each script with an executable: whereas a script foo would be simply placed in /path/to/venv/bin/ on POSIX, on Windows, the files foo.exe and foo-script.py (or foo-script.pyw) are placed in \path\to\venv\Scripts. The user can run \path\to\venv\Scripts\foo just as on POSIX.

The foo.exe file is just a copy of a stock launcher executable which finds its name from the C argv[0], and based in that name (foo in this case), invokes foo-script.py or foo-script.pyw with the appropriate Python interpreter.

I don't understand the relationship between this "stock launcher" and the PEP 397 launcher. They seem to have quite distinct requirements without much overlap. Specifically, I'm not aware that the current PEP 397 implementation could perform the same role as the "stock launcher" - IIUC, it has no special handling of the "-script" suffix or special logic based around its argv[0]. So unless I'm mistaken, even with PEP 397 accepted, either this "stock launcher" is still necessary anyway or the PEP398 launcher would need the addition of new features so it could replace the stock launcher. FWIW, Vinay and I exchanged some private mail recently about how to best integrate the PEP397 launcher with virtualenvs - and while we both agreed it would be nice, we couldn't come up with anything worthwhile. Having the launcher be aware of a virtualenv when invoked via file associations is problematic - for example, Windows Explorer is unlikely to have a virtualenv configured in its environment. Even when considering just command-line usage there are some edge-cases that make things problematic (eg, what if a script in a venv nominates a specific Python version via a shebang line? What if a venv is activated but the user/launcher attempts to execute a script outside the venv? etc.) On 29/05/2012 9:24 AM, Nick Coghlan wrote: ...

It would have been better if the issue of script management on Windows had been raised in PEP 405 itself - I likely would have declared PEP 397 a dependency *before* accepting it (even if that meant the feature missed the alpha 4 deadline and first appeared in beta 1, or potentially even missed 3.3 altogether).

However, I'm not going to withdraw the acceptance of the PEP over this - while I would have made a different decision at the time given the additional information (due to the general preference to treat Windows as a first class deployment target), I think reversing my decision now would make the situation worse rather than better.

That means the important question is what needs to happen before beta 1 at the end of June. As I see it, we have two ways forward:

1. My preferred option: bring PEP 397 up to scratch as a specification for the behaviour of the Python launcher (perhaps with Vinay stepping up as a co-author to help Mark if need be), find a BDFL delegate (MvL? Brian Curtin?) and submit that PEP for acceptance within the next few weeks. The updated PEP 397 should include an explanation of exactly how it will help with the correct implementation of PEP 405 on Windows (this may involve making the launcher pyvenv aware).

As above, it isn't clear to me how the additional complexity and list of caveats in real use make it worthwhile to have the PEP397 launcher pyvenv aware.

2. The fallback option: remove the currently checked in build artifacts from source control and incorporate them into the normal Windows build processes (both the main VS 2010 process, and at least the now-legacy VS 2008 process)

For alpha 4, I suggest going with MvL's suggestion - drop the binaries from Mercurial and accept that this aspect of PEP 405 simply won't work on Windows until the first beta.

Agreed - ISTM that this stock launcher is probably going to need to co-exist with the PEP397 launcher for the long term. Cheers, Mark

On 05/28/2012 04:24 PM, Nick Coghlan wrote:

It would have been better if the issue of script management on Windows had been raised in PEP 405 itself - I likely would have declared PEP 397 a dependency *before* accepting it (even if that meant the feature missed the alpha 4 deadline and first appeared in beta 1, or potentially even missed 3.3 altogether).

However, I'm not going to withdraw the acceptance of the PEP over this - while I would have made a different decision at the time given the additional information (due to the general preference to treat Windows as a first class deployment target), I think reversing my decision now would make the situation worse rather than better.

I think it's unfortunate that this issue (which is http://bugs.python.org/issue12394) has become entangled with PEP 405 at all, since AFAICT it is entirely orthogonal. This is a distutils2/packaging issue regarding how scripts are installed on Windows. It happens to be relevant when trying to install things into a PEP 405 venv on Windows, but it applies to a non-virtual Python installation on Windows every bit as much as it applies to a PEP 405 environment. In an earlier discussion with Vinay I thought we had agreed that it was an orthogonal issue and that this proposed patch for it would be removed from the PEP 405 reference implementation before it was merged to CPython trunk; I think that would have been preferable. This is why there is no mention of the issue in PEP 405 - it doesn't belong there, because it is not related.

That means the important question is what needs to happen before beta 1 at the end of June. As I see it, we have two ways forward:

1. My preferred option: bring PEP 397 up to scratch as a specification for the behaviour of the Python launcher (perhaps with Vinay stepping up as a co-author to help Mark if need be), find a BDFL delegate (MvL? Brian Curtin?) and submit that PEP for acceptance within the next few weeks. The updated PEP 397 should include an explanation of exactly how it will help with the correct implementation of PEP 405 on Windows (this may involve making the launcher pyvenv aware).

2. The fallback option: remove the currently checked in build artifacts from source control and incorporate them into the normal Windows build processes (both the main VS 2010 process, and at least the now-legacy VS 2008 process)

For alpha 4, I suggest going with MvL's suggestion - drop the binaries from Mercurial and accept that this aspect of PEP 405 simply won't work on Windows until the first beta.

Regardless, these sound like the right options moving forward, with the clarification that it is not any "aspect of PEP 405" that will not work until a fix is merged, it is simply an existing limitation of distutils2/packaging on Windows. And that if anything needs to be reverted, temporarily or permanently, it should not be all of the PEP 405 implementation, rather just this packaging fix. Carl

Nick Coghlan <ncoghlan <at> gmail.com> writes:

In that case: Vinay, please revert everything from the pyvenv commit that was actually related to issue #12394 rather than being part of the PEP 405 implementation. As Carl says, it's an unrelated change that needs to be discussed separately.

There's nothing in there related to #12394 - just a pysetup3.exe file, which I had originally overlooked and have now removed. Regards, Vinay Sajip

Am 29.05.2012 01:15, schrieb martin@v.loewis.de:

...

Comments welcome, especially on whether Windows users agree that something like this is needed in the absence of PEP 397 in Python 3.3.

AFAICT, there is no need to check in the binary into revision control. Instead, the Windows build process should create, package, and deploy them, and venv should then just expect that they are there.

So I request that this checkin is reverted, preferably before the alpha release.

I also agree with the fundamental principle that an open source project should never ever include binaries for which it doesn't also provide source code. If you cannot release the sources right now, do not release the binaries either.

Agreed. Vinay, please either let me know when this is rectified (see also Nick's request about reverting #12394 specific parts of the commit), or revert the whole PEP 405 implementation for now, if the time is too short: I don't want to delay the alpha much longer. There is still time until beta after all. Georg