On Tue, Jul 31, 2018 at 4:15 PM Victor Stinner wrote:

I finished my work on the _PyCoreConfig structure:

\o/ Thanks for all the good work!

Right now, the new API is still private. Nick Coghlan splitted the initialization in two parts: "core" and "main". I'm not sure that this split is needed.

The "core" config is basically the config for the runtime. In fact, PEP 432 renamed "core" to "runtime". Please keep the firm distinction between the runtime and the (main) interpreter.

We should see what to do, but it would be nice to make the _PyCoreConfig API public! IMHO it's way better than the old way to configuration Python initialization.

+1 However, shouldn't that happen after PEP 432 is accepted?

[snip]

The problem is that Python 3.7 got the half-baked implementation, and it caused issues:

* Calling Py_Main() after Py_Initialize() fails with a fatal error on Python 3.7.0 https://bugs.python.org/issue34008 * PYTHONOPTIMIZE environment variable is ignored by Py_Initialize() https://bugs.python.org/issue34247

I fixed the first issue, I'm now working on the second one to see how it can be fixed. Other option would be to backport the code from master to the 3.7 branch, since the code in master has a way better design. But it requires to backport a lot of changes. I'm not sure yet what is the best option.

Backporting shouldn't be so risky since it's all private API and there are few other changes in the relevant code since 3.7, right? It depends on if Ned's okay with it or not. :) -eric

2018-08-02 1:18 GMT+02:00 Eric Snow :

Backporting shouldn't be so risky since it's all private API and there are few other changes in the relevant code since 3.7, right? It depends on if Ned's okay with it or not. :)

I'm still doing further bug fixes and cleanup in the master branch: https://bugs.python.org/issue34170 I'm doing more and more changes. I just added two new files: Include/coreconfig.h and Python/coreconfig.h. IMHO it's better to put similar code in separated files. FYI Python/coreconfig.c has 954 files and Include/coreconfig.h has 319 lines. I'm ok to rename the structure and the files if you prefer a different name. About that, I'm working on a subproject: abandon Py_xxx global configuration variables to replace them with accessing interpreter->core_config->xxx. I'm not sure yet if it's a good idea or not, but it would allow to have two interpreters to have their own different configuration. Imagine two interpreters with different sys.path running in isolated mode. Or maybe an interpreter without importlib? One of the issue is that we have now two copies of the same option. For example, Py_BytesWarningFlag and interpreter->core_config->bytes_warning. That's why I would like to switch to core_config. But I'm also trying to make sure that the two variables have the same value: https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c246... Victor

2018-08-02 17:17 GMT+02:00 Eric Snow :

Note that there are backward compatibility issues to deal with. AFAIU if we start ignoring those global variables during initialization then it's going to cause problems for embedders.

One of the first operation of Py_Initialize(), Py_Main() and _PyCoreConfig_Read() is to get the current value of all global configuration variables. The change is more that modifying a global configuration variable after Py_Initialize() or Py_Main() may or may not have an effect. And in the future, it should no longer have effect. In short, these variables should only be read to populate the initialization configuration and then it should no longer change.

So I'm guessing you planned on deprecating any use of those global variables and, in the spirit of your goals for the C-API, provide a public API for extensions to access the info in the runtime config instead.

There is no *need* to deprecate anything. _PyCoreConfig remains fully compatible with them and there are now unit tests to make sure that their value is read at Python startup. The priority is: core config > global vars > env vars.

...

But I'm also trying to make sure that the two variables have the same value: https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c246...

Yep. That is necessary while the global config variables still exist. It's just risky since it's easy for us to change the config but forget to update the global config vars (that are shadowing the config). It would probably be worth making sure we have tests that verify that the two stay synchronized.

If possible, I would prefer that the configuration is *not* modified after Python has been initialized. I even hesitate to mark PyInterpreterState.core_config a constant to prevent such change. The idea would be to know exactly how Python has been initialize, to make the initialization more deterministic and explicit. To come back to a concrete example: https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c246... We can easily modify core_config->inspect before Python initialization. For this commit, it's just that I wanted to make tiny and incremental changes. Victor

Before I dive in, I'll say that I'd really like to hear Nick's opinion on all this. :) On Thu, Aug 2, 2018 at 9:59 AM Victor Stinner wrote:

2018-08-02 1:18 GMT+02:00 Eric Snow :

...

The "core" config is basically the config for the runtime. In fact, PEP 432 renamed "core" to "runtime". Please keep the firm distinction between the runtime and the (main) interpreter.

There is already something called _PyRuntime but it's shared between all interpreters.

_PyRuntime is a static global of type PyRuntimeState. It is where I consolidated (nearly) all the global runtime state last September.

_PyCoreConfig is already *per* interpreter.

This was done as part of the PEP 432 implementation, which I landed during PyCon 2017. If PyRuntimeState had existed already I'm sure it would be there instead.

Would you mind to elaborate what you mean by the "main interpreter"? I don't see anything obvious in the current code about what is a "main interpreter". Technically, I don't see anything like that.

The main interpreter is the first one created (during runtime initialization). It is special for a variety of reasons. Here are the ones I could think of: 1. the "main" thread will always belong to the main interpreter since it is the first PyThreadState created 2. runtime initialization uses the main interpreter exclusively 3. the first phase of runtime initialization (pre-initialization) ends with the main interpreter being *partially* configured 4. during the second phase (initializing), the partially-configured main interpreter facilitates the use of most of the C-API and may be used by embedders * this is the only time that an interpreter may be used in this way, and it only happens with the main interpreter 5. runtime finalization takes place using the main interpreter 6. the main interpreter is the last one destroyed during finalization 7. the REPL runs only in the main interpreter 8. the Python CLI is run in the main interpreter (i.e. in its __main__ module) 9. the main interpreter cannot be destroyed (except during finalization) 10. in Python code the main interpreter will always exist 11. it is the parent of all subinterpreters created in Python code (via PEP 554) 12. signals are handled only in the main interpreter 13. all single-threaded Python code is run in the main interpreter Note that there isn't anything special to the interpreter itself, but rather in where and how it's used. However, that matters and the runtime needs to treat it specially. I expect all this isn't well-documented because it is relevant to very few people.

I'm still not convinced that we need _PyMainInterpreterConfig:

Let's step back a moment and consider the course of events: 1. PEP 432 was created nearly 6 years ago to address the tangle that runtime initialization had become, with the intent of helping both the CPython maintainers and embedders 2. Nick did some re-organization around then (e.g. factoring out pylifecycle.c) to facilitate an implementation of the PEP 3. Nick implemented PEP 432, with a plan to merge it as a *private* API regardless of whether or not the PEP was accepted (with general consensus that doing so was a good idea) * see https://bitbucket.org/ncoghlan/cpython_sandbox/branch/pep432_modular_bootstr... * landing the private API would allow us to iron out the details of the PEP * work happened in spurts in 2013, 2015, and 2016; I kept poking Nick because the implementation was a big blocker for my multi-core/subinterpreters project 4. leading up to (and at) PyCon 2017, I forked Nick's branch, moved it to github, rebased it onto master, got it working again, created a PR, and finally landed it 5. since then the implementation has changed a bunch (due to Victor's much appreciate efforts) and has diverged from the PEP * notably it's unclear that code (especially pymain) strictly conforms to the phases in the PEP At this point the PEP is out of date. There have been several mailing list threads (all python-dev, IIRC) and some BPO issues where Victor solicited clarification or expressed a desire to change things and Nick gave feedback. None of that made it into the PEP. :( Consequently the PEP is inconsistent with the actual target. Furthermore, as was intended, we've learned of a few ways that the PEP could be improved. We *really* need to get the PEP updated so we can be sure everyone has all the info. Regarding the justification for the "main interpreter" config, the implementation has diverged from the original intent of the PEP: * the core/runtime config was meant to hold the minimal data needed to bootstrap/initialize the basic (limited) functionality of the C-API, including a restricted main interpreter + the struct members were strictly C plain-old-types since using PyObject would require the runtime to already be (partially) initialized + in the last year a lot of data has been added to this config; I don't know how much is strictly necessary to bootstrap the runtime (end of phase 1) and how much could be dealt with in phase 2 * the "main interpreter" config was meant to hold all the config needed to finish initializing the runtime (end of phase 2) + the struct members were mostly PyObject* (possible since most builtin types are available at this point) + the PEP proposes a bunch more fields than the implementation has; we planned on adding them a few at a time

_PyCoreConfig contains the same information. Is it really worth it to duplicate all _PyCoreConfig (more than 36 fields) in _PyMainInterpreterConfig? _PyMainInterpreterConfig adds a third copy of many paramters: another opportunity to introduce an inconsistency.

TBH, the PEP *should* have a clear answer for your question here, Victor. It has some explanation, but clearly it is incomplete (hence this continuing email thread). The duplication is partly a consequence of what has happened in the last year: a bunch of fields were added to the core config that were not in the PEP. However, note the key differences between the two structs: * core/runtime config + minimal + simple C fields + meant for embedders/pymain to bootstrap a limited runtime + not really meant to be used after calling Py_InitializeRuntime (AKA Py_InitializeCore) * main interpreter config + includes everything needed to finish full runtime initialization + has PyObject* fields + meant for embedders/pymain to finish initializing the runtime + not really meant to be used after calling Py_ConfigureMainInterpreter (except when initializing a subintepreter) Originally there wasn't much overlap. Furthermore, both of them are kept around so that, via the C-API (or directly in the CPython impl.), we could expose what data was used to initialize the runtime. This fills much the same role as the existing global Py_* variables. The duplication is due to there being C and PyObject versions. It is for the sake of embedders (and a little bit of sanity). The big reason why it shouldn't be a problem is because PyMainInterpreterConfig is generated directly from PyRuntimeConfig (AKA PyCoreConfig) and only *after* we've used the runtime config to bootstrap the limited runtime (after which it shouldn't be modified ever). So there's no risk of inconsistency, right? Perhaps it would make sense to only keep a const copy of both, to avoid modification?

Right now, an interpreter contains both: core and main configurations...

As noted above, the core/runtime config should probably be on PyRuntimeState instead. Regarding the "main" config, PyMainInterpreterConfig probably makes more sense as one of the following: 1. on PyRuntimeState, like the core/runtime config (since it's a one-off) 2. on PyInterpreterState, like now, but set to NULL on all but the main interpreter (which would allow us to distinguish the main interpreter from the rest) Both would require PyInterpreterConfig from PEP 432, but expanded to cover all config that might be unique to an interpreter. Also, conceptually there's a different between the-config-used-to-finish-runtime-init and the config-used-to-initialize-an-interpreter (including the main interpreter). In fact, PEP 432 does include a PyInterpreterConfig. However, in the current implementation, PyMainInterpreterConfig fills that role exclusively, which is confusing since we use the "main interpreter" config to initialize all interpreters (not just the main one). So here's what might make sense to do: 1. rename "core" to "runtime" (to reduce confusion) 2. move PyInterpreterState.runtime_config to PyRuntimeState.config + prevent modification after Py_InitializeRuntime() is called (e.g. keep a const copy)? 3. move PyInterpreterState.config to PyRuntimeConfig.main_config + prevent modification after Py_ConfigureMainInterpreter() is called (e.g. keep a const copy)? + keep the PyMainInterpreterConfig and Py_ConfigureMainInterpreter names 4. add PyInterpreterConfig with only the parts of PyMainInterpreterConfig needed to initialize any interpreter + add Py_NewInterpreterEx(PyInterpreterConfig) to allow explicitly passing a config? 5. add PyInterpreterState.config (type PyInterpreterConfig) to record the config used to initialize that interpreter + prevent modification after the interpreter is initialized (e.g. keep a const copy)?

I propose to *remove* _PyMainInterpreterConfig and rename _PyCoreConfig as _PyInterpreterConfig. I would also propose to merge again Py_Initialize() to have a single step instead of the current core step + main step: 2 steps.

So you are not in favor of PEP 432 then. :) -eric

On 1 August 2018 at 08:14, Victor Stinner wrote:

Hi,

I finished my work on the _PyCoreConfig structure: it's a C structure in Include/pystate.h which has many fields used to configure Python initialization. In Python 3.6 and older, these parameters were scatted around the code, and it was hard to get an exhaustive list of it.

This work is linked to the Nick Coghlan's PEP 432 "Restructuring the CPython startup sequence": https://www.python.org/dev/peps/pep-0432/

Right now, the new API is still private. Nick Coghlan splitted the initialization in two parts: "core" and "main". I'm not sure that this split is needed.

It is, because one of the aims is to make it clear when frozen bytecode (ala importlib) and other builtin modules can start to be used as part of the configuration process. That point is when the core configuration completes. By contrast, main interpreter configuration is only complete when you have full filesystem access as well (including external imports). That separation also means that an embedding application can choose *not* to proceed to the second step, and thus limit imports to the modules built in to the application.

We should see what to do, but it would be nice to make the _PyCoreConfig API public! IMHO it's way better than the old way to configuration Python initialization.

Step 1 will be to bring PEP 432 up to date with what actually happened - we learned a lot from your and Eric's efforts in actually implementing the draft design as a private API, but the current PEP still reflects the original design concept. Thanks for all your work on this! It's exciting to see it finally coming to fruition :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 3 August 2018 at 01:59, Victor Stinner wrote:

2018-08-02 1:18 GMT+02:00 Eric Snow :

...

The "core" config is basically the config for the runtime. In fact, PEP 432 renamed "core" to "runtime". Please keep the firm distinction between the runtime and the (main) interpreter.

There is already something called _PyRuntime but it's shared between all interpreters.

_PyCoreConfig is already *per* interpreter.

Would you mind to elaborate what you mean by the "main interpreter"? I don't see anything obvious in the current code about what is a "main interpreter". Technically, I don't see anything like that.

I'm still not convinced that we need _PyMainInterpreterConfig: _PyCoreConfig contains the same information. Is it really worth it to duplicate all _PyCoreConfig (more than 36 fields) in _PyMainInterpreterConfig? _PyMainInterpreterConfig adds a third copy of many paramters: another opportunity to introduce an inconsistency. Right now, an interpreter contains both: core and main configurations...

The issue is massive scope creep in the "core config": currently you need to fully specify everything in order to even use the builtin data structs. That's not the design goal of PEP 432: the idea is to have an absolutely bare minimum set of settings that gives a working C API, but won't let you access the host operating system. I wasn't reigning you in on it because there were real problems to be solved in getting the multi-stage start up to work at all given the current code structure. Now that it's working though, we should be looking to move settings back out of coreconfig, and reducing the amount of startup work that needs to be done using raw C code that can't rely on the CPython C API. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia