After helping Brett with the migration to importlib in 3.3, and looking at some of the ideas kicking around for additional CPython features that would affect the startup sequence, I've come to the conclusion that what we have now simply isn't sustainable long term. It's already the case that if you use certain options (specifically -W or -X), the interpreter will start accessing the C API before it has called Py_Initialize(). It's not cool when other people do that (we'd never accept code that behaved that way as a valid reproducer for a bug report), and it's *definitely* not cool that we're doing it (even though we seem to be getting away with it for the moment, and have been for a long time). The attached PEP is a first attempt at a plan for doing something about it. (My notes at http://wiki.python.org/moin/CPythonInterpreterInitialization provide additional context - let me know if you think there's more material on that page that should be in the PEP itself) The PEP is also available online at http://www.python.org/dev/peps/pep-0432/ Cheers, Nick. PEP: 432 Title: Simplifying the CPython startup sequence Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Dec-2012 Python-Version: 3.4 Post-History: 28-Dec-2012 Abstract ======== This PEP proposes a mechanism for simplifying the startup sequence for CPython, making it easier to modify the initialisation behaviour of the reference interpreter executable, as well as making it easier to control CPython's startup behaviour when creating an alternate executable or embedding it as a Python execution engine inside a larger application. Proposal Summary ================ This PEP proposes that CPython move to an explicit 2-phase initialisation process, where a preliminary interpreter is put in place with limited OS interaction capabilities early in the startup sequence. This essential core remains in place while all of the configuration settings are determined, until a final configuration call takes those settings and finishes bootstrapping the interpreter immediately before executing the main module. As a concrete use case to help guide any design changes, and to solve a known problem where the appropriate defaults for system utilities differ from those for running user scripts, this PEP also proposes the creation and distribution of a separate system Python (``spython``) executable which, by default, ignores user site directories and environment variables, and does not implicitly set ``sys.path[0]`` based on the current directory or the script being executed. Background ========== Over time, CPython's initialisation sequence has become progressively more complicated, offering more options, as well as performing more complex tasks (such as configuring the Unicode settings for OS interfaces in Python 3 as well as bootstrapping a pure Python implementation of the import system). Much of this complexity is accessible only through the ``Py_Main`` and ``Py_Initialize`` APIs, offering embedding applications little opportunity for customisation. This creeping complexity also makes life difficult for maintainers, as much of the configuration needs to take place prior to the ``Py_Initialize`` call, meaning much of the Python C API cannot be used safely. A number of proposals are on the table for even *more* sophisticated startup behaviour, such as better control over ``sys.path`` initialisation (easily adding additional directories on the command line in a cross-platform fashion, as well as controlling the configuration of ``sys.path[0]``), easier configuration of utilities like coverage tracing when launching Python subprocesses, and easier control of the encoding used for the standard IO streams when embedding CPython in a larger application. Rather than attempting to bolt such behaviour onto an already complicated system, this PEP proposes to instead simplify the status quo *first*, with the aim of making these further feature requests easier to implement. Key Concerns ============ There are a couple of key concerns that any change to the startup sequence needs to take into account. Maintainability --------------- The current CPython startup sequence is difficult to understand, and even more difficult to modify. It is not clear what state the interpreter is in while much of the initialisation code executes, leading to behaviour such as lists, dictionaries and Unicode values being created prior to the call to ``Py_Initialize`` when the ``-X`` or ``-W`` options are used [1_]. By moving to a 2-phase startup sequence, developers should only need to understand which features are not available in the core bootstrapping state, as the vast majority of the configuration process will now take place in that state. By basing the new design on a combination of C structures and Python dictionaries, it should also be easier to modify the system in the future to add new configuration options. Performance ----------- CPython is used heavily to run short scripts where the runtime is dominated by the interpreter initialisation time. Any changes to the startup sequence should minimise their impact on the startup overhead. (Given that the overhead is dominated by IO operations, this is not currently expected to cause any significant problems). The Status Quo ============== Much of the configuration of CPython is currently handled through C level global variables:: Py_IgnoreEnvironmentFlag Py_HashRandomizationFlag _Py_HashSecretInitialized _Py_HashSecret Py_BytesWarningFlag Py_DebugFlag Py_InspectFlag Py_InteractiveFlag Py_OptimizeFlag Py_DontWriteBytecodeFlag Py_NoUserSiteDirectory Py_NoSiteFlag Py_UnbufferedStdioFlag Py_VerboseFlag For the above variables, the conversion of command line options and environment variables to C global variables is handled by ``Py_Main``, so each embedding application must set those appropriately in order to change them from their defaults. Some configuration can only be provided as OS level environment variables:: PYTHONHASHSEED PYTHONSTARTUP PYTHONPATH PYTHONHOME PYTHONCASEOK PYTHONIOENCODING Additional configuration is handled via separate API calls:: Py_SetProgramName() (call before Py_Initialize()) Py_SetPath() (optional, call before Py_Initialize()) Py_SetPythonHome() (optional, call before Py_Initialize()???) Py_SetArgv[Ex]() (call after Py_Initialize()) The ``Py_InitializeEx()`` API also accepts a boolean flag to indicate whether or not CPython's signal handlers should be installed. Finally, some interactive behaviour (such as printing the introductory banner) is triggered only when standard input is reported as a terminal connection by the operating system. Also see more detailed notes at [1_] Proposal ======== (Note: details here are still very much in flux, but preliminary feedback is appreciated anyway) Core Interpreter Initialisation ------------------------------- The only configuration that currently absolutely needs to be in place before even the interpreter core can be initialised is the seed for the randomised hash algorithm. However, there are a couple of settings needed there: whether or not hash randomisation is enabled at all, and if it's enabled, whether or not to use a specific seed value. The proposed API for this step in the startup sequence is:: void Py_BeginInitialization(Py_CoreConfig *config); Like Py_Initialize, this part of the new API treats initialisation failures as fatal errors. While that's still not particularly embedding friendly, the operations in this step *really* shouldn't be failing, and changing them to return error codes instead of aborting would be an even larger task than the one already being proposed. The new Py_CoreConfig struct holds the settings required for preliminary configuration:: typedef struct { int use_hash_seed; size_t hash_seed; } Py_CoreConfig; To "disable" hash randomisation, set "use_hash_seed" and pass a hash seed of zero. (This seems reasonable to me, but there may be security implications I'm overlooking. If so, adding a separate flag or switching to a 3-valued "no randomisation", "fixed hash seed" and "randomised hash" option is easy) The core configuration settings pointer may be NULL, in which case the default behaviour of randomised hashes with a random seed will be used. A new query API will allow code to determine if the interpreter is in the bootstrapping state between core initialisation and the completion of the initialisation process:: int Py_IsInitializing(); While in the initialising state, the interpreter should be fully functional except that: * compilation is not allowed (as the parser is not yet configured properly) * The following attributes in the ``sys`` module are all either missing or ``None``: * ``sys.path`` * ``sys.argv`` * ``sys.executable`` * ``sys.base_exec_prefix`` * ``sys.base_prefix`` * ``sys.exec_prefix`` * ``sys.prefix`` * ``sys.warnoptions`` * ``sys.flags`` * ``sys.dont_write_bytecode`` * ``sys.stdin`` * ``sys.stdout`` * The filesystem encoding is not yet defined * The IO encoding is not yet defined * CPython signal handlers are not yet installed * only builtin and frozen modules may be imported (due to above limitations) * ``sys.stderr`` is set to a temporary IO object using unbuffered binary mode * The ``warnings`` module is not yet initialised * The ``__main__`` module does not yet exist <TBD: identify any other notable missing functionality> The main things made available by this step will be the core Python datatypes, in particular dictionaries, lists and strings. This allows them to be used safely for all of the remaining configuration steps (unlike the status quo). In addition, the current thread will possess a valid Python thread state, allow any further configuration data to be stored on the interpreter object rather than in C process globals. Any call to Py_BeginInitialization() must have a matching call to Py_Finalize(). It is acceptable to skip calling Py_EndInitialization() in between (e.g. if attempting to read the configuration settings fails) Determining the remaining configuration settings ------------------------------------------------ The next step in the initialisation sequence is to determine the full settings needed to complete the process. No changes are made to the interpreter state at this point. The core API for this step is:: int Py_ReadConfiguration(PyObject *config); The config argument should be a pointer to a Python dictionary. For any supported configuration setting already in the dictionary, CPython will sanity check the supplied value, but otherwise accept it as correct. Unlike Py_Initialize and Py_BeginInitialization, this call will raise an exception and report an error return rather than exhibiting fatal errors if a problem is found with the config data. Any supported configuration setting which is not already set will be populated appropriately. The default configuration can be overridden entirely by setting the value *before* calling Py_ReadConfiguration. The provided value will then also be used in calculating any settings derived from that value. Alternatively, settings may be overridden *after* the Py_ReadConfiguration call (this can be useful if an embedding application wants to adjust a setting rather than replace it completely, such as removing ``sys.path[0]``). Supported configuration settings -------------------------------- At least the following configuration settings will be supported:: raw_argv (list of str, default = retrieved from OS APIs) argv (list of str, default = derived from raw_argv) warnoptions (list of str, default = derived from raw_argv and environment) xoptions (list of str, default = derived from raw_argv and environment) program_name (str, default = retrieved from OS APIs) executable (str, default = derived from program_name) home (str, default = complicated!) prefix (str, default = complicated!) exec_prefix (str, default = complicated!) base_prefix (str, default = complicated!) base_exec_prefix (str, default = complicated!) path (list of str, default = complicated!) io_encoding (str, default = derived from environment or OS APIs) fs_encoding (str, default = derived from OS APIs) skip_signal_handlers (boolean, default = derived from environment or False) ignore_environment (boolean, default = derived from environment or False) dont_write_bytecode (boolean, default = derived from environment or False) no_site (boolean, default = derived from environment or False) no_user_site (boolean, default = derived from environment or False) <TBD: at least more from sys.flags need to go here> Completing the interpreter initialisation ----------------------------------------- The final step in the process is to actually put the configuration settings into effect and finish bootstrapping the interpreter up to full operation:: int Py_EndInitialization(PyObject *config); Like Py_ReadConfiguration, this call will raise an exception and report an error return rather than exhibiting fatal errors if a problem is found with the config data. After a successful call, Py_IsInitializing() will be false, while Py_IsInitialized() will become true. The caveats described above for the interpreter during the initialisation phase will no longer hold. Stable ABI ---------- All of the APIs proposed in this PEP are excluded from the stable ABI, as embedding a Python interpreter involves a much higher degree of coupling than merely writing an extension. Backwards Compatibility ----------------------- Backwards compatibility will be preserved primarily by ensuring that Py_ReadConfiguration() interrogates all the previously defined configuration settings stored in global variables and environment variables. One acknowledged incompatiblity is that some environment variables which are currently read lazily may instead be read once during interpreter initialisation. As the PEP matures, these will be discussed in more detail on a case by case basis. The Py_Initialize() style of initialisation will continue to be supported. It will use the new API internally, but will continue to exhibit the same behaviour as it does today, ensuring that sys.argv is not set until a subsequent PySys_SetArgv call. A System Python Executable ========================== When executing system utilities with administrative access to a system, many of the default behaviours of CPython are undesirable, as they may allow untrusted code to execute with elevated privileges. The most problematic aspects are the fact that user site directories are enabled, environment variables are trusted and that the directory containing the executed file is placed at the beginning of the import path. Currently, providing a separate executable with different default behaviour would be prohibitively hard to maintain. One of the goals of this PEP is to make it possible to replace much of the hard to maintain bootstrapping code with more normal CPython code, as well as making it easier for a separate application to make use of key components of ``Py_Main``. Including this change in the PEP is designed to help avoid acceptance of a design that sounds good in theory but proves to be problematic in practice. One final aspect not addressed by the general embedding changes above is the current inaccessibility of the core logic for deciding between the different execution modes supported by CPython: * script execution * directory/zipfile execution * command execution ("-c" switch) * module or package execution ("-m" switch) * execution from stdin (non-interactive) * interactive stdin <TBD: concrete proposal for better exposing the __main__ execution step> Implementation ============== None as yet. Once I have a reasonably solid plan of attack, I intend to work on a reference implementation as a feature branch in my BitBucket sandbox [2_] References ========== .. [1] CPython interpreter initialization notes (http://wiki.python.org/moin/CPythonInterpreterInitialization) .. [2] BitBucket Sandbox (https://bitbucket.org/ncoghlan/cpython_sandbox) Copyright =========== This document has been placed in the public domain. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia