[New-bugs-announce] [issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

I added a _PyCoreConfig structure to Python 3.7 which contains almost all parameters used to configure Python. Problems: _PyCoreConfig uses bytes and Unicode strings (char* and wchar_t*) whereas it is also used to setup the memory allocator and (filesystem, locale and stdio) encodings.

I propose to add a new _PyPreConfig which is the "strict minimum" configuration to setup encodings and the memory allocator. In practice, it also contains parameters which directly or indirectly impacts the allocator and encodings. For example, isolated impacts use_environment which impacts the allocator (PYTHONMALLOC environment variable). Another example: dev_mode=1 sets the allocator to "debug".

The command line arguments are now parsed twice. _PyPreConfig only parses a few parameters like -E, -I and -X. A temporary _PyPreCmdline is used to store command line arguments like -X options.

I moved structures closer to where they are used. "Global" _PyMain structure has been removed. _PyCmdline now lives way shorter than previously and is moved from main.c to coreconfig.c. The idea is to better control when and how memory is allocated.

In term of API, we get something like:

    _PyCoreConfig config = _PyCoreConfig_INIT;
    config.preconfig.stdio_encoding = "iso8859-1";
    config.preconfig.stdio_errors = "replace";
    config.user_site_directory = 0;
    ...

    _PyInitError err = _Py_InitializeFromConfig(&config);
    if (_Py_INIT_FAILED(err)) {
        _Py_ExitInitError(err);
    }
    ...
    Py_Finalize();
    return 0;

"config.preconfig.stdio_errors" syntax isn't great, but it's simpler to implement than duplicating all _PyPreConfig fields into _PyCoreConfig.

----------
components: Interpreter Core
messages: 336791
nosy: vstinner
priority: normal
severity: normal
status: open
title: Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36142>
_______________________________________