[Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)

Eric Snow ericsnowcurrently at gmail.com
Thu Aug 2 11:17:49 EDT 2018

On Thu, Aug 2, 2018 at 3:50 AM Victor Stinner <vstinner at redhat.com> wrote:
> I'm still doing further bug fixes and cleanup in the master branch:
> https://bugs.python.org/issue34170
> I'm doing more and more changes.

Yeah, it's a question of what you plan to backport.  As Barry
suggested, it would be great if you had a WIP PR for the backport,
just so Ned (and others) has a point of reference.

> I just added two new files:
> Include/coreconfig.h and Python/coreconfig.h. IMHO it's better to put
> similar code in separated files.
> FYI Python/coreconfig.c has 954 files and Include/coreconfig.h has 319 lines.

Nick might have a better opinion, particularly when in comes to a C
codebase, but I'm in favor of keeping separate things separate, and
especially when in relates to the runtime.  (Historically we haven't
been great about considering the runtime as a distinct part of
CPython.)  Hence +1 to keeping the runtime config separate, especially
given the size of the files. :)  Presumably pystate.c (and pystate.h)
got smaller by roughly the same amount?

Also, would it make sense for at least some of coreconfig.h to live in
Include/internal, either as coreconfig.h or in the internal pystate.h?

> I'm ok to rename the structure and the files if you prefer a different name.

I'd love to see all the "core" names changed to "runtime", in the same
way that PEP 432 was updated.  It was a point of confusion in the PEP
until we changed the name, and doing so helped.  I thought we had also
changed the code, but apparently not.

For that matter, I'd love to see PEP 432 and the codebase synced up.
While the overall plan is still consistent, a number of details (e.g.
the intent and content of the "core" config) have diverged.

> About that, I'm working on a subproject: abandon Py_xxx global
> configuration variables to replace them with accessing
> interpreter->core_config->xxx.

+1  IMHO it's a natural consequence of having a runtime/core config.
In fact, having both is problematic because it's easy for us to
accidentally ignore a global-var-as-config (prior to Py_Initialize);
the relationship between those global variables and runtime
initialization isn't very obvious in the code (unlike the config
struct).  It's also confusing for embedders if we have both.  As I've
already expressed, I'm definitely in favor of improving encapsulation
of the runtime (and moving away from globals). :)

Note that there are backward compatibility issues to deal with.  AFAIU
if we start ignoring those global variables during initialization then
it's going to cause problems for embedders.  If we get rid of the
variables altogether then it would break extension modules that
currently rely on reading those parts of the runtime config.  So I'm
guessing you planned on deprecating any use of those global variables
and, in the spirit of your goals for the C-API, provide a public API
for extensions to access the info in the runtime config instead.

FWIW, I recall that Nick and I discussed this relative to PEP 432 a
while ago and remember the decision to stay with the status quo for
now (to avoid scope creep in the PEP).  Apparently that consideration
did not get recorded in the PEP (that I could see with a quick skim of
the text).  The mailing lists may have the discussion somewhere.

> I'm not sure yet if it's a good idea or
> not, but it would allow to have two interpreters to have their own
> different configuration. Imagine two interpreters with different
> sys.path running in isolated mode. Or maybe an interpreter without
> importlib?

Yeah, a number of interesting possibilities open up as we further
encapsulate the runtime and move away from globals.

> One of the issue is that we have now two copies of the same option.
> For example, Py_BytesWarningFlag and
> interpreter->core_config->bytes_warning. That's why I would like to
> switch to core_config.


> But I'm also trying to make sure that the two variables have the same value:
> https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c2461e92d

Yep.  That is necessary while the global config variables still exist.
It's just risky since it's easy for us to change the config but forget
to update the global config vars (that are shadowing the config).  It
would probably be worth making sure we have tests that verify that the
two stay synchronized.


More information about the Python-Dev mailing list