<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#330033" bgcolor="#ffffff">
On 3/19/2011 7:38 PM, Mark Hammond wrote:
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">Thanks
for the feedback!
<br>
</blockquote>
<br>
And thanks for more complete explanations.<br>
<br>
Sadly I was offline when writing my first response, and couldn't
view the man page for execve you referred to. Having just read it,
I think it would be total gibberish to a Windows user who has only
learned Python 2.x programming, and is now faced with migrating to
3.x, and wants to use this launcher to help make that a gradual
process, instead of the full, instant migration required by the
installation of Python 3.x wiping out the associations from Python
2.x leaving his system with 2.x unusable (from a naïve point of
view). I have not found any support either from the installer or
the documentation (but maybe I missed some things) that tell how to
make Python 2.x and 3.x coexist happily on Windows, short of simply
reinstalling the one you want to use.<br>
<br>
A Windows user who has only learned Python 2.x programming would not
necessarily have ever heard of execve, would not realize execve(2)
means it is from the 2nd chapter of the Unix man pages, meaning an
API call, would not know C programming, would not know what #include
<unistd.h> means, would not understand syntax like "const" or
"[]". The description is a bit friendlier, if they get that far,
but they may still be quite confused by wondering how to create and
provide argv and envp in the right forms, wondering what SIGCHLD,
PID, set-user-ID bit, SIG_IGN, SIG_DFL, SIGTRAP, set-group-ID,
effective id, a.out, binary executable, shared library stubs,
ld.so(8), ELF executable, PT_INTERP, and /lib/ld-unix.so.1 (and .2)
are. <br>
<br>
In short, anyone that is a Unix C programmer can easily understand
this stuff, but anyone that is a Windows Python-only programmer will
feel like understanding the launcher by reading that man page is
similar in effort to getting their masters degree, and that if it is
that hard to slowly migrate their 300 Python scripts to use Python
3.x while still using Python 2.x for those that are not converted,
that they might as well just stick with Python 2.x.<br>
<br>
Yes, the argument could be made that parts of Python also require
some Unix knowledge and basic C knowledge to use effectively... but
there are many Python programs that can be written without using the
parts that require that knowledge.<br>
<br>
That said, I don't think it would be an onerous task to extract a
reasonable description of #! usage from the execve man page, and
eliminate the Unix masters degree requirement. In fact, only the
following two paragraph fragments seem to be relevant to #! lines:<br>
<br>
<blockquote type="cite">
... a script starting with a line of the form "<b>#! <i>interpreter</i></b>
[arg]". ... the interpreter must be a
valid pathname for an executable which is not itself a script,
which will be invoked as <b>interpreter</b> [arg] <i>filename</i>.
<br>
<p>A maximum line length of 127 characters is allowed for the
first line in a #! executable shell script.</p>
</blockquote>
<br>
More below in context...<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">On
19/03/2011 7:44 PM, Glenn Linderman wrote:
<br>
<blockquote type="cite">Not all of the ideas below are
complementary to each other, some are
<br>
either or, to allow different thoughts to be inspired or
different
<br>
directions to be taken.
<br>
<br>
Thanks for starting a PEP.
<br>
<br>
On 3/18/2011 11:02 PM, Mark Hammond wrote:
<br>
<br>
<blockquote type="cite"> The launcher should be as simple as
possible (but no simpler.)
<br>
</blockquote>
<br>
The launcher could be simpler if it isn't used for launching
interactive
<br>
interpreters as well as script references via Windows
associations (more
<br>
about that after the next quote).
<br>
<br>
The launcher could be simpler if it just read through the file
of its
<br>
first parameter until it finds a line starting with #@ (process
as a
<br>
Windows version of Unix #!) or starting without a # (error
case). This
<br>
avoids the need parse such lines.
<br>
</blockquote>
<br>
I don't agree with that at all. The rules for the shebang parsing
are very simple - see the man-page I reference in the pep - the
file must start with the characters '#!' and the entire command
must be < 127 chars. The only real "parsing" needed is to
check if the specified command starts with one of 2 fixed
strings. I believe this is simpler than parsing multiple lines of
the file.
<br>
</blockquote>
<br>
Ah, but Mark! You are confusing (and maybe I do in some of my
comments too) the complexity of the launcher versus the complexity
of describing the launcher. It is really the complexity of
describing the launcher (and remember, Unix doesn't need a launcher,
it has #!, so you need to describe it to Windows users for Windows
users) that matters most. I find I often rewrite portions of my
programs when I write the documentation to make them easier to
describe. And some say the docs should be written first, but then I
find that I have to rewrite the docs to make the code possible :)
Whatever the order, the goal should be to make the program as useful
as possible to a variety of users, with as little complexity as
possible in the documentation.<br>
<br>
So I still claim that it is easier to tell a Windows user that the
launcher looks at comment lines starting from line 1 until it finds
a line starting with #@ or finds a non-comment line, and executes it
like the first paragraph I extracted from execve above, than to
explain about how Unix works, why Unix uses / instead of \, why
there is a special case for /usr/bin/python and /usr/bin/env python,
and why there is a 127 character limit (which need not be the case
for a Windows specific line). After describing that, the only
reference to Unix that is necessary is to say that Unix implements a
similar feature[footnote to execve man page] in the operating system
with a line starting with #! which must be the very first line and
limited to 127 characters, and that this Windows launcher is more
flexible to accommodate using both techniques for cross-platform
scripts.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<br>
> Remember, the typical
<br>
<blockquote type="cite">Windows user is not likely to place a #!
line in their scripts in the
<br>
first place, so teaching them what a Unix #! line is, and how
the
<br>
parameter after it should be something that Windows doesn't even
use,
<br>
and the launcher has to work hard to interpret, is not as simple
as
<br>
possible.
<br>
</blockquote>
<br>
I disagree with various aspects of that - the "typical Windows
user" is not going to add a shebang or variation of to their
Python source files, period. </blockquote>
<br>
IMO, when faced with migrating from 2.x to 3.x incrementally, the
"typical Windows user" will look to the Python provided launcher
solution to help (presuming that there is such a thing). If it
doesn't require a Unix masters degree, they'll give it whirl.
Something like<br>
<br>
Install the launcher feature from Python 3.3 (or available
separately somewhere), and add<br>
#@C:\Python3.3\python.exe before the first non-comment line in the
new Python 3 scripts.<br>
<br>
In fact, this description just inspired me to suggest that the
launcher should launch a Python 2 if it cannot find a #@ (or #! if I
can't convince you of #@) line... that way old scripts stick with
Python 2 until they are converted.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">IMO,
the kind of user who would is already somewhat likely to know of
the shebang convention, so it would not be foreign. Those
remaining who are not familiar with it can simply be pointed at
existing docs etc for the shebang line and their new knowledge now
works on more than Windows (and more than Python on non-Windows
platforms)
<br>
</blockquote>
<br>
I agree that the users that already understand #! would gladly add
it to their Windows Python scripts and understand it. And I agree
that those are probably the only users that would understand the
execve Unix masters degree well enough to figure out a launcher that
is described by pointed to that man page. But I think a simple
description of a launcher that uses Windows terminology (Unix users
don't need a launcher) would be acceptable to a much larger subset
of Windows users, that have learned a bit of Python scripting. And
those are the ones that need it. The ones that understand #! have
also figured out Windows assoc and ftype baloney, and have figured
out how to switch back and forth from one version of Python (or
anything else). Or have given up using XXXX.py and simply always
invoke scripts with C:\PythonX.y\python.exe XXXX.py in frustration
at the lack of a Python aid in this area.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<br>
<blockquote type="cite">The launcher could be simpler if the
Python installer placed versioned
<br>
Python executables on the PATH. Unfortunately, historically it
hasn't.
<br>
If it did, would, or the launcher installer would place them
there for
<br>
pre-existing versions of Python, then the launcher could work by
<br>
launching the appropriate version of Python, expecting Windows
to find
<br>
it on the PATH. The PEP doesn't address the level of internal
complexity
<br>
of the launcher necessary to find where Python has been
installed,
<br>
neither for CPython, nor for the alternate implementations to be
supported.
<br>
</blockquote>
<br>
The PEP intentionally doesn't, but the implementation does - it
already does exactly that for CPython. Other implementation may
need a different strategy, but we can cross that bridge when we
come to it.
<br>
</blockquote>
<br>
The PEP could therefore be clearer by discussing the CPython
implementation strategy, even if it points out that other
implementations may need a different strategy. Putting PythonX.Y on
the PATH eliminates the need for the launchers (py.exe and pyw.exe)
to be on the path. They are then only needed in the Windows
associations, as I suggested somewhere.<br>
<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<br>
<blockquote type="cite">The launcher could be simpler if a
directory \usr\bin were created under
<br>
Windows Program Files, placed on the PATH, and %ProgramFiles%
prepended
<br>
to the Unix #! line, with the Python/Jython/Cython installers
placing
<br>
appropriately versioned executables in that directory. Could
even start
<br>
a trend for programs ported from Unix. One could even place an
"env"
<br>
program there, for more simplicity.
<br>
</blockquote>
<br>
Again, I disagree - I think in practice the code would be more
complex, and having Python assert it "owns" such directories and
executables is a can of worms we should avoid.
<br>
</blockquote>
<br>
This was a random thought I had, which went a "more Unix" direction,
instead of the "less Unix" direction of #@. I didn't particularly
like this one either, but it seemed to be a possible alternative, so
I didn't want it to go unmentioned, in case thousands of other
Python users thought it would be a great idea if only they had
thought of it... :)<br>
<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<blockquote type="cite">
<blockquote type="cite"> * When used to launch an
interactive Python interpreter, the launcher
<br>
will support the first command-line argument optionally
be a
<br>
version specifier in the form "-n[.n]" (where n is a
single
<br>
integer) to nominate a specific version be used. For
example,
<br>
while "py.exe" may locate and launch the latest Python
2.x
<br>
implementation installed, a command-line such as
"py.exe -3" could
<br>
specify the latest Python 3.x implementation be
launched, while
<br>
"py.exe -2.6" could specify Python 2.6 be located and
launched.
<br>
If a Python 2.x implementation is desired to be
launched with the
<br>
-3 flag, the command-line would need to be similar to
"py.exe -2
<br>
-3" (or the specific version of Python could obviously
be
<br>
launched manually without use of this launcher.)
<br>
</blockquote>
<br>
I think that a python launcher that is "on the PATH" that could
be used
<br>
to launch an interactive Python, should be different than one
that is
<br>
used to launch XXXX.py[w] scripts.
<br>
</blockquote>
<br>
I believe that if you know the Python you want is already on the
PATH, you should just use 'python.exe' instead of 'py.exe'. I
don't see any reason to use this launcher for interactive Python
sessions where this is the case.
<br>
<br>
<blockquote type="cite">1) python should be invoked interactively
by typing "python" or
<br>
"pythonX[.Y]" at the CMD prompt, not "py". This can be done
without a
<br>
launcher, if appropriate versioned pythons are placed on the
PATH. The
<br>
launcher is really and only needed for handling XXXX.py[w]
scripts,
<br>
which, in the Windows way of thinking, can only be associated
with one
<br>
specific, system-wide configured version of Python (presently,
the
<br>
latest one wins). The script itself is not examined to modify
such an
<br>
association. The Unix !# line provides such modification on
Unix.
<br>
</blockquote>
<br>
OK, I think we found something we can agree on :) If the
python.exe you want is on your path, just ignore this launcher for
anything other than file associations.
<br>
</blockquote>
<br>
So I think we are agreeing here, mostly. And you mentioned having
the CPython implementation PythonX.Y get on the PATH somehow. So
just keep the python launcher off the PATH, remove its then
unnecessary option syntax, and we agree totally :)<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<blockquote type="cite">2) If the launcher provides command line
options for the "benefit" of
<br>
launching interactive Python interpreters, those command line
options
<br>
can have data puns with script names, or can conflict with
actual Python
<br>
options. I believe Python 2 already has a -3 option, for
example. And
<br>
Windows users are not trained that "-" introduces an option
syntax, but
<br>
rather "/". Most _programmer_ users would probably be aware of
"-" as an
<br>
option syntax, but Python is used as a language for
non-programmers in
<br>
some circles, and few Windows non-programmers understand "/"
much less
<br>
"-" and not even command lines very well. So not using a
launcher for
<br>
launching interactive Python sidesteps all that: Python itself
is
<br>
introduced and taught, and no need to teach about (or even have)
<br>
launcher options that could possibly conflict and confuse, in
addition
<br>
to Python options that may conflict with script names already.
(I have
<br>
seen lots of Windows users use leading punctuation characters on
<br>
filenames to affect sort order and grouping of files in My
Documents,
<br>
not knowing they can create subdirectories/subfolders, or not
wanting to
<br>
bother with them, since all their applications default to
opening things
<br>
from My Documents.)
<br>
</blockquote>
<br>
I'm not 100% sure of the points you are trying to make above, but
the gist of it seems to be the same as (1) - use python.exe
directly if you prefer - in which case I agree. Obviously if
people choose to use the new launcher interactively they are doing
so because they see some benefit and would be willing to
understand how it works. So I don't see any problems here as I'm
not advocating it would replace "python.exe" in interactive
scenarios except where the user actively chooses to.
<br>
</blockquote>
<br>
Does anyone see any benefit to using a launcher, if PythonX.Y is on
the PATH? I don't. I see putting PythonX.Y as an extremely useful
feature available for Unix Python that is totally missing from
Windows Python, to its large detriment. But that is not a feature
the launcher should have to supply, but since Python hasn't been, a
launcher installer may need to, to compensate.<br>
<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<br>
<blockquote type="cite">3) Unix !# lines can have embedded options
after the program name on the
<br>
line. Such options would be another source of potential conflict
with
<br>
launcher options, if the launcher has options for use with
launching
<br>
interactive interpreters.
<br>
<br>
Item 3 is also an issue for the PEP even apart from its use as
an
<br>
interactive Python launcher; since options may exist on the Unix
#!
<br>
line, a discussion of how and if they are handled by the
launcher should
<br>
be included in the PEP.
<br>
</blockquote>
<br>
I believe a reference to the execve man-page and the note in the
PEP that we will support the description there, including
limitations, is enough - but I'm happy to change this if people
agree it is underspecified or confusing.
<br>
</blockquote>
<br>
Per the beginning of this email, I think you need to stay far away
from referencing the execve man page in a description of a Windows
launcher, except perhaps as a footnote.<br>
<br>
Until there is a benefit seen to using a launcher from the command
line, I think it should (1) not be on the PATH (2) have no option
syntax, which gets things back to a simple description.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<br>
<blockquote type="cite">
<br>
<blockquote type="cite"> * Environment varialbes will be
used to override the semantics for
<br>
determining exactly what version of Python will be
used. For
<br>
example, while a shebang line of "/usr/bin/python2"
will
<br>
automatically locate a Python 2.x implementation, an
environment
<br>
variable can override exactly which Python 2.x
implementation will
<br>
be chosen. Similarly for "/usr/bin/python" etc.
<br>
</blockquote>
<br>
Clarify if environment variables can be used to override
semantics for
<br>
shebang lines of the form "/usr/bin/python2.x".
<br>
</blockquote>
<br>
On re-reading the PEP, I notice I deleted something which is
important on the mistaken belief the execve man page would make it
clear. Only the first command-line argument will be checked for a
shebang line. If the first argument is an option (ie, starts with
a '-'), nothing is examined for a shebang line. This is what the
reference implementation does and I'll be sure to update the PEP
to reflect this.
<br>
<br>
The PEP does state the optional -N.N argument must be the first
argument. Therefore, the presence of the -N.N argument will avoid
any shebang processing at all so can't impact the version selected
via the shebang line. This makes sense to me as someone
explicitly executing "py.exe -3 foo.py" is explicitly overriding
the version they want to run the script with.
<br>
</blockquote>
<br>
Not using the launcher from the command line, and it not having any
available options, would sidestep this issue nicely.<br>
<br>
Here you didn't respond to the question about overriding semantics
of /usr/bin/pythonX.Y, but your reply to Dj did. Please clarify it
in the PEP, however. Thanks.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<blockquote type="cite">If alternate implementations are to be
supported, additional virtual
<br>
commands will be required, not just these two. Each one adds
complexity
<br>
to the launcher.
<br>
</blockquote>
<br>
This is true - each implementation would need custom code to sniff
out the requested implementation version. I don't think this is a
burden though and can be worked around by using fully-qualified
executable names in the shebang line.
<br>
</blockquote>
<br>
No, the workaround of using fully-qualified executable names in the
shebang line makes the script now dysfunctional on Unix. So it
would not be possible to use CPython on Unix, and IronPython on
Windows, nor jpython on Unix and CPython on Windows, nor any other
cross-implementation scripts, should different implementations be
used on different platforms in a particular environment. This is
one of the reasons I like a separate #@ line for a Windows
launcher. It permits the flexibility of specifying both the
implementation and the version separately for the different
platforms, which would be complex to achieve if both platforms look
at the same #! line.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">
<br>
<blockquote type="cite">
<blockquote type="cite"> Non-virtual shebang lines should be
discouraged as they make the
<br>
script specific to a specific Windows installation.
However, they
<br>
are supported for maximum flexibility.
<br>
</blockquote>
<br>
This is a false statement.
<br>
</blockquote>
<br>
I believe it is a true statement - needing words like "likely" and
"most" to dispute it just demonstrates that IMO. Note that I
don't dispute your statements, but instead believe that "most"
simply isn't good enough - I want "always".
<br>
</blockquote>
<br>
I don't think you'll get "always", because of environments that use
different implementations on different platforms. To get always,
you need the flexibility to specify implementation and version for
each platform separately... two lines. I recall an environment I
once worked in where yacc was used on Unix, and bison on Windows,
for example... as a counter-argument for anyone that wants to jump
in a say that such a split implementation environment would surely
never exist. And differing C implementations between Unix/Windows
also, in another environment. I wasn't using Python in those days,
but I recall having different versions of Perl available between
Unix/Windows. Etc.<br>
<br>
It may be true that it is better to rely on the Windows path,
specifying #@pythonX.Y instead of #@C:\PythonX.Y\python.exe if
Python, or the launcher installer, can arrange to get all the
pythonX.Y on the PATH.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">Unfortunately,
it is clear we disagree on some fundamental points and I can't see
a way to change my PEP that would keep both you and I happy.
Therefore, the only reasonable resolution would be for you to
draft a competing PEP and reference implementation which
python-dev can informally vote on.
<br>
</blockquote>
<br>
Let's enumerate... and maybe if we keep talking we'll increase the
agreement list and decrease the disagreement list.<br>
<br>
New ideas in this message:<br>
1) If launcher doesn't find a #!/#@ line, it should run python2 to
support scripts without such lines.<br>
<br>
Agreements:<br>
1) Python needs PythonX.Y to be on the PATH<br>
2) Need a launcher to handle Windows associations that otherwise
only allow one version for one extension<br>
3) Need a separate launcher for .py and .pyw<br>
4) Support for as many environments as possible, and as many
installations as possible for each environment.<br>
<br>
Disagreements:<br>
1) #! only, with "virtual" paths versus #! for Unix, #@ for
Windows to support different implementations and versions on the
different platforms.<br>
2) Use launcher from command line versus use PythonX.Y from the
command line for interactive Python interpreters, don't put launcher
on the PATH, its interactive option just adds complexity, not
benefit. (Not sure if we disagree all that much here, my position is
very close to +/- 0 on this one -- just wondering if there is any
benefit to the added complexity of documenting it for command line
use)<br>
3) Reference execve man page and expect Windows users to understand
it versus document one paragraph for how a similar feature works
on Windows<br>
4) The launcher should be as simple as possible (but no simpler)
versus The launcher should be as useful as possible to a variety of
users, with as little complexity as possible in the documentation.<br>
<br>
What else? Can I borrow and tweak your reference implementation?
I'm seriously awash in multiple projects, most of which want to use
cross-platform Python, so I see this topic as extremely important
for my future projects, but haven't written C in so long I'd hate to
try to commit to more than a reference implementation in Python.<br>
<br>
<blockquote cite="mid:4D8568B4.3070102@skippinet.com.au" type="cite">Cheers,
<br>
<br>
Mark
<br>
</blockquote>
<br>
Hopefully you are still cheery :)<br>
</body>
</html>