[Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Glenn Linderman]

On 3/19/2011 7:38 PM, Mark Hammond wrote:
> Thanks for the feedback!

And thanks for more complete explanations.

Sadly I was offline when writing my first response, and couldn't view 
the man page for execve you referred to.  Having just read it, I think 
it would be total gibberish to a Windows user who has only learned 
Python 2.x programming, and is now faced with migrating to 3.x, and 
wants to use this launcher to help make that a gradual process, instead 
of the full, instant migration required by the installation of Python 
3.x wiping out the associations from Python 2.x leaving his system with 
2.x unusable (from a naïve point of view).  I have not found any support 
either from the installer or the documentation (but maybe I missed some 
things) that tell how to make Python 2.x and 3.x coexist happily on 
Windows, short of simply reinstalling the one you want to use.

A Windows user who has only learned Python 2.x programming would not 
necessarily have ever heard of execve, would not realize execve(2) means 
it is from the 2nd chapter of the Unix man pages, meaning an API call, 
would not know C programming, would not know what #include <unistd.h> 
means, would not understand syntax like "const" or "[]".  The 
description is a bit friendlier, if they get that far, but they may 
still be quite confused by wondering how to create and provide argv and 
envp in the right forms, wondering what SIGCHLD, PID, set-user-ID bit, 
SIG_IGN, SIG_DFL, SIGTRAP, set-group-ID, effective id, a.out, binary 
executable, shared library stubs, ld.so(8), ELF executable, PT_INTERP, 
and /lib/ld-unix.so.1 (and .2) are.

In short, anyone that is a Unix C programmer can easily understand this 
stuff, but anyone that is a Windows Python-only programmer will feel 
like understanding the launcher by reading that man page is similar in 
effort to getting their masters degree, and that if it is that hard to 
slowly migrate their 300 Python scripts to use Python 3.x while still 
using Python 2.x for those that are not converted, that they might as 
well just stick with Python 2.x.

Yes, the argument could be made that parts of Python also require some 
Unix knowledge and basic C knowledge to use effectively... but there are 
many Python programs that can be written without using the parts that 
require that knowledge.

That said, I don't think it would be an onerous task to extract a 
reasonable description of #! usage from the execve man page, and 
eliminate the Unix masters degree requirement.  In fact, only the 
following two paragraph fragments seem to be relevant to #! lines:

> ... a script starting with a line of the form "*#! /interpreter/* 
> [arg]".  ... the interpreter must be a valid pathname for an 
> executable which is not itself a script, which will be invoked as 
> *interpreter* [arg] /filename/.
>
> A maximum line length of 127 characters is allowed for the first line 
> in a #! executable shell script.
>

More below in context...

> On 19/03/2011 7:44 PM, Glenn Linderman wrote:
>> Not all of the ideas below are complementary to each other, some are
>> either or, to allow different thoughts to be inspired or different
>> directions to be taken.
>>
>> Thanks for starting a PEP.
>>
>> On 3/18/2011 11:02 PM, Mark Hammond wrote:
>>
>>>      The launcher should be as simple as possible (but no simpler.)
>>
>> The launcher could be simpler if it isn't used for launching interactive
>> interpreters as well as script references via Windows associations (more
>> about that after the next quote).
>>
>> The launcher could be simpler if it just read through the file of its
>> first parameter until it finds a line starting with #@ (process as a
>> Windows version of Unix #!) or starting without a # (error case). This
>> avoids the need parse such lines.
>
> I don't agree with that at all.  The rules for the shebang parsing are 
> very simple - see the man-page I reference in the pep - the file must 
> start with the characters '#!' and the entire command must be < 127 
> chars.  The only real "parsing" needed is to check if the specified 
> command starts with one of 2 fixed strings.  I believe this is simpler 
> than parsing multiple lines of the file.

Ah, but Mark!  You are confusing (and maybe I do in some of my comments 
too) the complexity of the launcher versus the complexity of describing 
the launcher.  It is really the complexity of describing the launcher 
(and remember, Unix doesn't need a launcher, it has #!, so you need to 
describe it to Windows users for Windows users) that matters most.  I 
find I often rewrite portions of my programs when I write the 
documentation to make them easier to describe.  And some say the docs 
should be written first, but then I find that I have to rewrite the docs 
to make the code possible :)  Whatever the order, the goal should be to 
make the program as useful as possible to a variety of users, with as 
little complexity as possible in the documentation.

So I still claim that it is easier to tell a Windows user that the 
launcher looks at comment lines starting from line 1 until it finds a 
line starting with #@ or finds a non-comment line, and executes it like 
the first paragraph I extracted from execve above, than to explain about 
how Unix works, why Unix uses / instead of \, why there is a special 
case for /usr/bin/python and /usr/bin/env python, and why there is a 127 
character limit (which need not be the case for a Windows specific 
line).  After describing that, the only reference to Unix that is 
necessary is to say that Unix implements a similar feature[footnote to 
execve man page] in the operating system with a line starting with #! 
which must be the very first line and limited to 127 characters, and 
that this Windows launcher is more flexible to accommodate using both 
techniques for cross-platform scripts.

>
> > Remember, the typical
>> Windows user is not likely to place a #! line in their scripts in the
>> first place, so teaching them what a Unix #! line is, and how the
>> parameter after it should be something that Windows doesn't even use,
>> and the launcher has to work hard to interpret, is not as simple as
>> possible.
>
> I disagree with various aspects of that - the "typical Windows user" 
> is not going to add a shebang or variation of to their Python source 
> files, period. 

IMO, when faced with migrating from 2.x to 3.x incrementally, the 
"typical Windows user" will look to the Python provided launcher 
solution to help (presuming that there is such a thing).  If it doesn't 
require a Unix masters degree, they'll give it whirl.  Something like

Install the launcher feature from Python 3.3 (or available separately 
somewhere), and add
#@C:\Python3.3\python.exe before the first non-comment line in the new 
Python 3 scripts.

In fact, this description just inspired me to suggest that the launcher 
should launch a Python 2 if it cannot find a #@ (or #! if I can't 
convince you of #@) line... that way old scripts stick with Python 2 
until they are converted.

> IMO, the kind of user who would is already somewhat likely to know of 
> the shebang convention, so it would not be foreign.  Those remaining 
> who are not familiar with it can simply be pointed at existing docs 
> etc for the shebang line and their new knowledge now works on more 
> than Windows (and more than Python on non-Windows platforms)

I agree that the users that already understand #! would gladly add it to 
their Windows Python scripts and understand it.  And I agree that those 
are probably the only users that would understand the execve Unix 
masters degree well enough to figure out a launcher that is described by 
pointed to that man page.  But I think a simple description of a 
launcher that uses Windows terminology (Unix users don't need a 
launcher) would be acceptable to a much larger subset of Windows users, 
that have learned a bit of Python scripting.  And those are the ones 
that need it.  The ones that understand #! have also figured out Windows 
assoc and ftype baloney, and have figured out how to switch back and 
forth from one version of Python (or anything else).  Or have given up 
using  XXXX.py and simply always invoke scripts with 
C:\PythonX.y\python.exe XXXX.py in frustration at the lack of a Python 
aid in this area.

>
>> The launcher could be simpler if the Python installer placed versioned
>> Python executables on the PATH. Unfortunately, historically it hasn't.
>> If it did, would, or the launcher installer would place them there for
>> pre-existing versions of Python, then the launcher could work by
>> launching the appropriate version of Python, expecting Windows to find
>> it on the PATH. The PEP doesn't address the level of internal complexity
>> of the launcher necessary to find where Python has been installed,
>> neither for CPython, nor for the alternate implementations to be 
>> supported.
>
> The PEP intentionally doesn't, but the implementation does - it 
> already does exactly that for CPython.  Other implementation may need 
> a different strategy, but we can cross that bridge when we come to it.

The PEP could therefore be clearer by discussing the CPython 
implementation strategy, even if it points out that other 
implementations may need a different strategy.  Putting PythonX.Y on the 
PATH eliminates the need for the launchers (py.exe and pyw.exe) to be on 
the path.  They are then only needed in the Windows associations, as I 
suggested somewhere.


>
>> The launcher could be simpler if a directory \usr\bin were created under
>> Windows Program Files, placed on the PATH, and %ProgramFiles% prepended
>> to the Unix #! line, with the Python/Jython/Cython installers placing
>> appropriately versioned executables in that directory. Could even start
>> a trend for programs ported from Unix. One could even place an "env"
>> program there, for more simplicity.
>
> Again, I disagree - I think in practice the code would be more 
> complex, and having Python assert it "owns" such directories and 
> executables is a can of worms we should avoid.

This was a random thought I had, which went a "more Unix" direction, 
instead of the "less Unix" direction of #@.  I didn't particularly like 
this one either, but it seemed to be a possible alternative, so I didn't 
want it to go unmentioned, in case thousands of other Python users 
thought it would be a great idea if only they had thought of it... :)


>>>      * When used to launch an interactive Python interpreter, the 
>>> launcher
>>>        will support the first command-line argument optionally be a
>>>        version specifier in the form "-n[.n]" (where n is a single
>>>        integer) to nominate a specific version be used.  For example,
>>>        while "py.exe" may locate and launch the latest Python 2.x
>>>        implementation installed, a command-line such as "py.exe -3" 
>>> could
>>>        specify the latest Python 3.x implementation be launched, while
>>>        "py.exe -2.6" could specify Python 2.6 be located and launched.
>>>        If a Python 2.x implementation is desired to be launched with 
>>> the
>>>        -3 flag, the command-line would need to be similar to "py.exe -2
>>>        -3" (or the specific version of Python could obviously be
>>>        launched manually without use of this launcher.)
>>
>> I think that a python launcher that is "on the PATH" that could be used
>> to launch an interactive Python, should be different than one that is
>> used to launch XXXX.py[w] scripts.
>
> I believe that if you know the Python you want is already on the PATH, 
> you should just use 'python.exe' instead of 'py.exe'.  I don't see any 
> reason to use this launcher for interactive Python sessions where this 
> is the case.
>
>> 1) python should be invoked interactively by typing "python" or
>> "pythonX[.Y]" at the CMD prompt, not "py". This can be done without a
>> launcher, if appropriate versioned pythons are placed on the PATH. The
>> launcher is really and only needed for handling XXXX.py[w] scripts,
>> which, in the Windows way of thinking, can only be associated with one
>> specific, system-wide configured version of Python (presently, the
>> latest one wins). The script itself is not examined to modify such an
>> association. The Unix !# line provides such modification on Unix.
>
> OK, I think we found something we can agree on :)  If the python.exe 
> you want is on your path, just ignore this launcher for anything other 
> than file associations.

So I think we are agreeing here, mostly.  And you mentioned having the 
CPython implementation PythonX.Y get on the PATH somehow.  So just keep 
the python launcher off the PATH, remove its then unnecessary option 
syntax, and we agree totally :)

>> 2) If the launcher provides command line options for the "benefit" of
>> launching interactive Python interpreters, those command line options
>> can have data puns with script names, or can conflict with actual Python
>> options. I believe Python 2 already has a -3 option, for example. And
>> Windows users are not trained that "-" introduces an option syntax, but
>> rather "/". Most _programmer_ users would probably be aware of "-" as an
>> option syntax, but Python is used as a language for non-programmers in
>> some circles, and few Windows non-programmers understand "/" much less
>> "-" and not even command lines very well. So not using a launcher for
>> launching interactive Python sidesteps all that: Python itself is
>> introduced and taught, and no need to teach about (or even have)
>> launcher options that could possibly conflict and confuse, in addition
>> to Python options that may conflict with script names already. (I have
>> seen lots of Windows users use leading punctuation characters on
>> filenames to affect sort order and grouping of files in My Documents,
>> not knowing they can create subdirectories/subfolders, or not wanting to
>> bother with them, since all their applications default to opening things
>> from My Documents.)
>
> I'm not 100% sure of the points you are trying to make above, but the 
> gist of it seems to be the same as (1) - use python.exe directly if 
> you prefer - in which case I agree.  Obviously if people choose to use 
> the new launcher interactively they are doing so because they see some 
> benefit and would be willing to understand how it works.  So I don't 
> see any problems here as I'm not advocating it would replace 
> "python.exe" in interactive scenarios except where the user actively 
> chooses to.

Does anyone see any benefit to using a launcher, if PythonX.Y is on the 
PATH?  I don't.  I see putting PythonX.Y as an extremely useful feature 
available for Unix Python that is totally missing from Windows Python, 
to its large detriment.  But that is not a feature the launcher should 
have to supply, but since Python hasn't been, a launcher installer may 
need to, to compensate.


>
>> 3) Unix !# lines can have embedded options after the program name on the
>> line. Such options would be another source of potential conflict with
>> launcher options, if the launcher has options for use with launching
>> interactive interpreters.
>>
>> Item 3 is also an issue for the PEP even apart from its use as an
>> interactive Python launcher; since options may exist on the Unix #!
>> line, a discussion of how and if they are handled by the launcher should
>> be included in the PEP.
>
> I believe a reference to the execve man-page and the note in the PEP 
> that we will support the description there, including limitations, is 
> enough - but I'm happy to change this if people agree it is 
> underspecified or confusing.

Per the beginning of this email, I think you need to stay far away from 
referencing the execve man page in a description of a Windows launcher, 
except perhaps as a footnote.

Until there is a benefit seen to using a launcher from the command line, 
I think it should (1) not be on the PATH (2) have no option syntax, 
which gets things back to a simple description.

>
>>
>>>      * Environment varialbes will be used to override the semantics for
>>>        determining exactly what version of Python will be used.  For
>>>        example, while a shebang line of "/usr/bin/python2" will
>>>        automatically locate a Python 2.x implementation, an environment
>>>        variable can override exactly which Python 2.x implementation 
>>> will
>>>        be chosen.  Similarly for "/usr/bin/python" etc.
>>
>> Clarify if environment variables can be used to override semantics for
>> shebang lines of the form "/usr/bin/python2.x".
>
> On re-reading the PEP, I notice I deleted something which is important 
> on the mistaken belief the execve man page would make it clear.  Only 
> the first command-line argument will be checked for a shebang line.  
> If the first argument is an option (ie, starts with a '-'), nothing is 
> examined for a shebang line.  This is what the reference 
> implementation does and I'll be sure to update the PEP to reflect this.
>
> The PEP does state the optional -N.N argument must be the first 
> argument.  Therefore, the presence of the -N.N argument will avoid any 
> shebang processing at all so can't impact the version selected via the 
> shebang line.  This makes sense to me as someone explicitly executing 
> "py.exe -3 foo.py" is explicitly overriding the version they want to 
> run the script with.

Not using the launcher from the command line, and it not having any 
available options, would sidestep this issue nicely.

Here you didn't respond to the question about overriding semantics of 
/usr/bin/pythonX.Y, but your reply to Dj did.  Please clarify it in the 
PEP, however.  Thanks.

>> If alternate implementations are to be supported, additional virtual
>> commands will be required, not just these two. Each one adds complexity
>> to the launcher.
>
> This is true - each implementation would need custom code to sniff out 
> the requested implementation version.  I don't think this is a burden 
> though and can be worked around by using fully-qualified executable 
> names in the shebang line.

No, the workaround of using fully-qualified executable names in the 
shebang line makes the script now dysfunctional on Unix.   So it would 
not be possible to use CPython on Unix, and IronPython on Windows, nor 
jpython on Unix and CPython on Windows, nor any other 
cross-implementation scripts, should different implementations be used 
on different platforms in a particular environment.  This is one of the 
reasons I like a separate #@ line for a Windows launcher.  It permits 
the flexibility of specifying both the implementation and the version 
separately for the different platforms, which would be complex to 
achieve if both platforms look at  the same #! line.

>
>>>      Non-virtual shebang lines should be discouraged as they make the
>>>      script specific to a specific Windows installation. However, they
>>>      are supported for maximum flexibility.
>>
>> This is a false statement.
>
> I believe it is a true statement - needing words like "likely" and 
> "most" to dispute it just demonstrates that IMO.  Note that I don't 
> dispute your statements, but instead believe that "most" simply isn't 
> good enough - I want "always".

I don't think you'll get "always", because of environments that use 
different implementations on different platforms.  To get always, you 
need the flexibility to specify implementation and version for each 
platform separately... two lines.  I recall an environment I once worked 
in where yacc was used on Unix, and bison on Windows, for example... as 
a counter-argument for anyone that wants to jump in a say that such a 
split implementation environment would surely never exist.  And 
differing C implementations between Unix/Windows also, in another 
environment. I wasn't using Python in those days, but I recall having 
different versions of Perl available between Unix/Windows.  Etc.

It may be true that it is better to rely on the Windows path, 
specifying  #@pythonX.Y  instead of  #@C:\PythonX.Y\python.exe if 
Python, or the launcher installer, can arrange to get all the pythonX.Y 
on the PATH.

> Unfortunately, it is clear we disagree on some fundamental points and 
> I can't see a way to change my PEP that would keep both you and I 
> happy. Therefore, the only reasonable resolution would be for you to 
> draft a competing PEP and reference implementation which python-dev 
> can informally vote on.

Let's enumerate... and maybe if we keep talking we'll increase the 
agreement list and decrease the disagreement list.

New ideas in this message:
1) If launcher doesn't find a #!/#@ line, it should run python2 to 
support scripts without such lines.

Agreements:
1) Python needs PythonX.Y to be on the PATH
2) Need a launcher to handle Windows associations that otherwise only 
allow one version for one extension
3) Need a separate launcher for .py and .pyw
4) Support for as many environments as possible, and as many 
installations as possible for each environment.

Disagreements:
1) #! only, with "virtual" paths   versus   #! for Unix, #@ for Windows 
to support different implementations and versions on the different 
platforms.
2) Use launcher from command line  versus   use PythonX.Y from the 
command line for interactive Python interpreters, don't put launcher on 
the PATH, its interactive option just adds complexity, not benefit. (Not 
sure if we disagree all that much here, my position is very close to +/- 
0 on this one -- just wondering if there is any benefit to the added 
complexity of documenting it for command line use)
3) Reference execve man page and expect Windows users to understand it  
versus   document one paragraph for how a similar feature works on Windows
4) The launcher should be as simple as possible (but no simpler)  
versus  The launcher should be as useful as possible to a variety of 
users, with as little complexity as possible in the documentation.

What else?  Can I borrow and tweak your reference implementation?  I'm 
seriously awash in multiple projects, most of which want to use 
cross-platform Python, so I see this topic as extremely important for my 
future projects, but haven't written C in so long I'd hate to try to 
commit to more than a reference implementation in Python.

> Cheers,
>
> Mark

Hopefully you are still cheery :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110320/8426ec53/attachment-0001.html>