[Python-Dev] PEP 432 progress: Python initalization
victor.stinner at gmail.com
Thu Dec 14 10:16:15 EST 2017
Serhiy Storchaka seems to be worried by the high numbers of commits in
https://bugs.python.org/issue32030 "PEP 432: Rewrite Py_Main()", so
let me explain the context of this work :-)
To prepare CPython to implement my UTF-8 Mode PEP (PEP 540), I worked
on the implementation of Nick Coghlan's PEP 432:
PEP 432 -- Restructuring the CPython startup sequence
The startup sequence is a big pile of code made of multiple functions:
main(), Py_Main(), Py_Initialize(), Py_Finalize()... and a lot of tiny
"configuration" functions like Py_SetPath().
Over the years, many configuration options were added in the middle of
the code. The priority of configuration options is not always correct
between command line options, envrionment variables, configuration
files (like "pyenv.cfg"), etc. For technical reasons, it's hard to
impement properly the -E option (ignore PYTHON* environment
For example, the new PYTHONCOERCECLOCALE environment variable (of PEP
538) doesn't handle properly -E (it ignores -E), because it was too
complex to support -E. -- I'm working on fixing this.
Last weeks, I mostly worked on the Py_Main() function,
Modules/getpath.c and PC/getpathp.c, to "refactor" the code:
* Split big functions (300 to 500 lines) into multiple small functions
(50 lines or less), to make it easily to follow the control flow and
to allow to more easily move code
* Replace static and global variables with memory allocated on the heap.
* Reorganize how the configuration is read: populate a first temporary
structure (_PyMain using wchar_t*), then create Python objects
(_PyMainInterpreterConfig) to finish with the real configuration (like
setting attributes of the sys module). The goal is to centralize all
code reading configuration to fix the priority and to simplify the
My motivation was to write a correct implementation of the UTF-8 Mode (PEP 540).
Nick's motivation is to make CPython easily to embed. His plan for
Python 3.8 is to give access to the new _PyCoreConfig and
_PyMainInterpreterConfig structures to:
* easily give access to most (if not all?) configuration options to "embedders"
* allow to configure Python without environment variables, command
line options, configuration files, but only using these structures
* allow to configure Python using Python objects (PyObject*) rather
than C types (like wchar_t*)
(I'm not sure that I understood correctly, so please read the PEP 432 ;-))
IMHO the most visible change of the PEP 432 is to split Python
initialization in two parts:
* Core: strict minimum to use the Python C API
* Main: everything else
The goal is to introduce the opportunity to configure Python between
Core and Main.
The implementation is currently a work-in-progress. Nick will not have
the bandwidth, neither do I, to update his PEP and finish the
implementation, before Python 3.7. So this work remains private until
at least Python 3.8.
Another part of the work is to enhance the documentation. You can for
example now find an explicit list of C functions which can be called
And also a list of functions that must not be called before
Py_Initialize(), whereas you might want to call them :-)
More information about the Python-Dev