PEP: 397 Title: Python launcher for Windows Version: $Revision$ Last-Modified: $Date$ Author: Mark Hammond Status: Draft Type: Standards Track Content-Type: text/plain Created: 15-Mar-2011 Post-History: 17-May-2011, 15-Mar-2011 Abstract This PEP describes a Python launcher for the Windows platform. A Python launcher is a single executable which uses a number of heuristics to locate a Python executable and launch it with a specified command line. Rationale Windows provides "file associations" so an executable can be associated with an extension, allowing for scripts to be executed directly in some contexts (eg., double-clicking the file in Windows Explorer.) Until now, a strategy of "last installed Python wins" has been used and while not ideal, has generally been workable due to the conservative changes in Python 2.x releases. As Python 3.x scripts are often syntactically incompatible with Python 2.x scripts, a different strategy must be used to allow files with a '.py' extension to use a different executable based on the Python version the script targets. This will be done by borrowing the existing practices of another operating system - scripts will be able to nominate the version of Python they need by way of a "shebang" line, as described below. Unix-like operating systems (referred to simply as "Unix" in this PEP) allow scripts to be executed as if they were executable images by examining the script for a "shebang" line which specifies the actual executable to be used to run the script. This is described in detail in the evecve(2) man page [1] and while user documentation will be created for this feature, for the purposes of this PEP that man page describes a valid shebang line. Additionally, these operating systems provide symbolic-links to Python executables in well-known directories. For example, many systems will have a link /usr/bin/python which references a particular version of Python installed under the operating-system. These symbolic links allow Python to be executed without regard for where Python it actually installed on the machine (eg., without requiring the path where Python is actually installed to be referenced in the shebang line or in the PATH.) PEP 394 'The "python" command on Unix-Like Systems' [2] describes additional conventions for more fine-grained specification of a particular Python version. These 2 facilities combined allow for a portable and somewhat predictable way of both starting Python interactively and for allowing Python scripts to execute. This PEP describes an implementation of a launcher which can offer the same benefits for Python on the Windows platform and therefore allows the launcher to be the executable associated with '.py' files to support multiple Python versions concurrently. While this PEP offers the ability to use a shebang line which should work on both Windows and Unix, this is not the primary motivation for this PEP - the primary motivation is to allow a specific version to be specified without inventing new syntax or conventions to describe it. An overview of the launcher. This PEP outlines the general functionality and key guidelines of a launcher for Windows. It is accompanied by a reference implementation [3], written in Python, which defines the detailed implementation. Over time, changes to the implementation may be desired - if the changes adhere to the guidelines in this PEP and have been made following the standard Python development model this PEP need not change. In other words, this PEP makes no attempt to describe in detail every feature offered by the launcher but instead to offer guidelines the launcher should adhere to. The actual implementation of the launcher will be written in C and will not link directly or indirectly with Python, but the Python based reference implementation should always remain the canonical description of the functionality. Any changes proposed to the launcher functionality should first be made and agreed to in the Python implementation, after which time the C implementation can be changed to reflect the new reference. Any deviations between the functionality of the reference implementation versus the C implementation due should imply the C implementation is in error (although some exceptions will need to be made due to the nature of the reference implementation being in Python) It is expected that 2 versions of the launcher will be needed - one which is a console program and one which is a "windows" (ie., GUI) program. These 2 launchers correspond to the 'python.exe' and 'pythonw.exe' executables which currently ship with Python. The console launcher will be named 'py.exe' and the Windows one named 'pyw.exe'. The "windows" (ie., GUI) version of the launcher will attempt to locate and launch pythonw.exe even if a virtual shebang line nominates simply "python" - infact, the trailing 'w' notation will not be supported in the virtual shebang line at all. The launcher will be distributed with all future versions of Python and if possible, should be installed somewhere likely to already be on the system PATH (eg., the Windows System32) directory. If installed, the "console" version of the launcher should be associated with .py files and the "windows" version associated with .pyw files. The launcher will not be tied to a specific version of Python - eg., a launcher distributed with Python 3.3 should be capable of locating and executing any Python 2.x and Python 3.x version. Future versions of the launcher should remain backwards compatible with older versions, so later versions of Python can install an updated version of the launcher without impacting how the previously installed version of the launcher is used. The launcher will offer some conveniences for Python developers working interactively - for example, starting the launcher with no command-line arguments will launch the default Python with no command-line arguments. Further, command-line arguments will be supported to allow a specific Python version to be launched interactively - however, these conveniences will not detract from the primary purpose of launching scripts and will be easy to avoid if desired. Guidelines for a Python launcher. The Python launcher described in this PEP will intentionally be constrained to the use-cases described in the Rationale section above. It will not attempt to be a general purpose script launcher or shebang processor. The launcher should support for format of shebang lines as described in [1], including all restrictions listed. The launcher should support shebang lines commonly found on Unix. For example, a shebang line of '!# /usr/bin/python' should work even though there is unlikely to be an executable in the relative Windows directory "\usr\bin". This means that many scripts can use a single shebang line and be likely to work on both Unix and Windows without modification. The launcher will support fully-qualified paths to executables. While this will make the script inherently non-portable, it is a feature offered by Unix and would be useful for Windows users in some cases. The launcher will be capable of supporting implementations other than CPython, such as jython and IronPython, but given both the absence of common links on Unix (such as "/usr/bin/jython") and the inability for the launcher to automatically locate the installation location of these implementations on Windows, the launcher will support this via customization options. Scripts taking advantage of this will not be portable (as these customization options must be set to reflect the configuration of the machine on which the launcher is running) but this ability is nonetheless considered worthwhile. On Unix, the user can control which specific version of Python is used by adjusting the links in /usr/bin to point to the desired version. As the launcher on Windows will not use Windows links, environment variables will be used to override the semantics for determining exactly what version of Python will be used. For example, while a shebang line of "/usr/bin/python2" will automatically locate a Python 2.x implementation, an environment variable can override exactly which Python 2.x implementation will be chosen. Similarly for "/usr/bin/python" and "/usr/bin/python3". This is specified in detail later in this PEP. Shebang line parsing If the first command-line argument does not start with a dash ('-') character, an attempt will be made to open that argument as a file and parsed for a shebang line according to the rules in [1]. Once parsed, the command will be categorized according to the following rules: * If the command starts with the definition of a customized command followed by a space character, the customized command will be used. See below for a description of customized commands. * If the command starts with the strings '/usr/bin/python', '/usr/bin/env python' or 'python' the command will be treated as a 'virtual command' and the rules described in Python Version Qualifiers (below) will be used to locate the executable to use. * Otherwise the command is assumed to be directly ready to execute - ie. a fully-qualified path optionally followed by arguments. The contents of the string will not be parsed - it will be passed directly to the Windows CreateProcess function after appending the name of the script and the launcher command-line arguments. This means that the rules used by CreateProcess will be used, including how relative path names and executable references without extensions are treated. Notably, the Windows command processor will not be used, so special rules used by the command processor (such as automatic appending of extensions other than '.exe', support for batch files, etc) will not be used. The use of 'virtual' shebang lines will be encouraged as this should allow for portable shebang lines to be specified which work on multiple operating systems and different installations of the same operating system. If the first argument can not be opened as a file or if no valid shebang line can be found, the launcher will act as if a shebang line of '!#python' was found - ie., a default Python interpreter will be located and the arguments passed to that. However, if a valid shebang line is found but the process specified by that line can not be started, the default interpreter will not be started - the error to create the specified child process will cause the launcher to display an appropriate message and terminate with a specific exit code. Virtual commands in shebang lines: Virtual Commands are shebang lines which start with the strings '/usr/bin/python', '/usr/bin/env python' or 'python'. Optionally, the virtual command may be suffixed with a version qualifier (see below), such as '/usr/bin/python2' or '/usr/bin/python3.2'. The command executed is based on the rules described in Python Version Qualifiers below. Customized Commands: The launcher will support the ability to define "Customized Commands" in a Windows .ini file (ie, a file which can be parsed by the Windows function GetPrivateProfileString). A section called '[commands]' can be created with key names defining the virtual command and the value specifying the actual command-line to be used for this virtual command. For example, if an INI file has the contents: [commands] vpython=c:\bin\vpython.exe -foo Then a shebang line of '#! vpython' in a script named 'doit.py' will result in the launcher using the command-line 'c:\bin\vpython.exe -foo doit.py' Two .ini files will be searched by the launcher - 'py.ini' in the current user's "application data" directory (ie, the directory returned by calling the Windows function SHGetFolderPath with CSIDL_APPDATA) and 'py.ini' in the same directory as the launcher. Commands specified in the "application directory" will have precendence over the one next to the executable, and the same .ini files will be used by both the Windows and Console versions of the launcher. Python Version Qualifiers Some of the features described allow an optional Python version qualifier to be used. A version qualifier starts with a major version number and can optionally be followed by a period ('.') and a minor version specifier. If the minor qualifier is specified, it may optionally be followed by "-32" to indicate the 32bit implementation of that version be used. Note that no "-64" qualifier is necessary as this is the default implementation (see below). If no version qualifiers are found, the environment variable PY_PYTHON can be set to specify the default version qualifier - the default value is "2". Note this value could specify just a major version (eg., "2") or a major.minor qualifier (eg., "2.6"), or even major.minor-32. If no minor version qualfiers are found, the environment variable PY_PYTHON{major} (where {major} is the current major version qualifier as determined above) can be set to specify the full version. If no such option is found, the launcher will enumerate the installed Python versions and use the latest minor release found for the major version, which is likely, although not guaranteed, to be the most recently installed version in that family. On 64bit Windows with both 32bit and 64bit implementations of the same (major.minor) Python version installed, the 64bit version will always be preferred. This will be true for both 32bit and 64bit implementations of the launcher - a 32bit launcher will prefer to execute a 64bit Python installation of the specified version if available. This is so the behavior of the launcher can be predicted knowing only what versions are installed on the PC and without regard to the order in which they were installed. As noted above, an optional "-32" suffix can be used on a version specifier to change this behaviour. Examples: * If no relevant options are set, the commands 'python' and 'python2' will use the latest Python 2.x version installed and the command 'python3' will use the lastest Python 3.x installed. * The commands 'python3.1' and 'python2.7' will not consult any options at all as the versions are fully specified. * if 'PY_PYTHON=3', the commands 'python' and 'python3' will both use the latest installed Python 3 version. * if 'PY_PYTHON=3.1-32', the command 'python' will use the 32bit implementation of 3.1 whereas the command 'python3' will use the latest installed Python (PY_PYTHON was not considered at all as a major version was specified.) * if 'PY_PYTHON=3' and 'PY_PYTHON3=3.1', the commands 'python' and 'python3' will both use specifically 3.1 In addition to environment variables, the same settings can be configured in the .INI file used by the launcher. The section in the INI file will be called '[defaults]' and the key name will be the same as the environment variables without the leading 'PY_' prefix (and note that the key names in the INI file are case insensitive. The contents of an environment variable will override things specified in the INI file. For example: * Setting 'PY_PYTHON=3.1' is equivilent to the INI file containing: [defaults] python=3.1 * Setting 'PY_PYTHON=3' and 'PY_PYTHON3=3.1' is equivilent to the INI file containing: [defaults] python=3 python3=3.1 Command-line handling Only the first command-line argument will be checked for a shebang line and only if that argument does not start with a '-'. If the only command-line argument is "-h" or "--help", the launcher will print a small banner and command-line usage, then pass the argument to the default Python. This will cause help for the launcher being printed followed by help for Python itself. The output from the launcher will attempt to clearly indicate the extended help information is coming from the launcher and not Python. As a concession to interactively launching Python, the launcher will support the first command-line argument optionally being a dash ("-") followed by a version qualifier, as described above, to nominate a specific version be used. For example, while "py.exe" may locate and launch the latest Python 2.x implementation installed, a command-line such as "py.exe -3" could specify the latest Python 3.x implementation be launched, while "py.exe -2.6-32" could specify a 32bit implementation Python 2.6 be located and launched. If a Python 2.x implementation is desired to be launched with the -3 flag, the command-line would need to be similar to "py.exe -2 -3" (or the specific version of Python could obviously be launched manually without use of this launcher.) Note that this feature can not be used with shebang processing as the file scanned for a shebang line and this argument must both be the first argument and therefore are mutually exclusive. All other arguments will be passed untouched to the child Python process. Process Launching Ideally, the launcher process would execute Python directly inside the same process, primarily so the parent of the launcher process could terminate the launcher and have the Python interpreter terminate. If the launcher executes Python as a sub-process and the parent of the launcher terminates the launcher, the Python process will be unaffected. However, there are a number of practical problems associated with this approach. Windows does not support the execv* family of Unix functions, so this could only be done by the launcher dynamically loading the Python DLL, but this would have a number of side-effects. The most serious side effect of this is that the value of sys.executable would refer to the launcher instead of the Python implementation. Many Python scripts use the value of sys.executable to launch child processes, and these scripts may fail to work as expected if the launcher is used. Consider a "parent" script with a shebang line of '#! /usr/bin/python3' which attempts to launch a child script (with no shebang) via sys.executable - currently the child is launched using the exact same version running the parent script. If sys.executable referred to the launcher the child would be likely executed using a Python 2.x version and would be likely to fail with a SyntaxError. A solution for this would need to be found before this could be considered. Another hurdle is the support for alternative Python implementations using the "customized commands" feature described above, where loading the command dynamically into a running executable is not possible. The final hurdle is the rules above regarding 64bit and 32bit programs - a 32bit launcher would be unable to load the 64bit version of Python and vice-versa. Given these considerations, the launcher will execute it's command in a child process, remaining alive while the child process is executing, then terminate with the same exit code as returned by the child. To address the concerns regarding the termination of the launcher not killing the child, the Win32 Job API will be used to arrange so that the child process is automatically killed when the parent is terminated (although children of that child process will continue as is the case now.) As this Windows API is available in Windows XP and later, this launcher will not work on Windows 2000 or earlier. References [1] http://linux.die.net/man/2/execve [2] http://www.python.org/dev/peps/pep-0394/ [3] http://bugs.python.org/issue11629 Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: