[Python-Dev] New Python Initialization API

[Nick Coghlan]

On Sat, 30 Mar 2019 at 12:45, Steve Dower <steve.dower at python.org> wrote:
>
> On 29Mar.2019 1830, Victor Stinner wrote:
> > The purpose of the PEP 587 is to have a working document so everyone
> > can look at the proposed API (stay focused to the API rather than
> > bothering with the implementation). IMHO it's now time to get more
> > people looking at the Python Initialization.
> >
> >> But there are enough of us
> >> with fuzzy but valid ideas in our heads that we really need that
> >> brainstorming session to mix them together and find something feasible.
> >> Maybe we're best to put it off until PyCon at this point.
> >
> > Python 3.8 feature freeze is scheduled at the end of May, less than
> > one month after the PyCon. It might be a little bit too late, no?
>
> I don't think we want to rush this in for 3.8 at this point anyway. The
> design of how Python is embedded is one of those things that could
> drastically affect the scenarios it gets used for in the future
> (probably half of my tasks at work right now involve embedding CPython),
> so I'd like to get it right.

Victor and I chatted about this, and I think it would be good to get
something in to Python 3.8 that gives applications embedding CPython
access to the same level of control that we have from the native
CPython CLI application - the long and the short of the *nix embedding
bug reports that have come in since Python 3.7 was released is that
locale coercion and UTF-8 mode don't quite work properly when an
application is relying solely on the Py_Initialize() and Py_Main()
APIs and doesn't have access to the extra preconfiguration steps that
have been added to get everything to work nicely together and avoid
mojibake in the native CPython CLI app. Victor's gone to great lengths
to try to make them work, but the unfortunate fact is that by the time
they're called, too many other things have often happened in the
embedding application for CPython to be able to get all the data
completely self-consistent.

Thus the two changes go hand in hand: reverting the old initialization
APIs back to their Python 3.6 behaviour to fix the embedding
regressions our two PEPs inadvertently introduced for some
applications when running in the POSIX locale, while also exposing new
initialization APIs so embedding apps can explicitly opt in to
behaving the same way as the CPython CLI does. Affected apps would
then switch to Python 3.8 at the earliest opportunity, and stop
supporting Python 3.7 as the embedded Python version.

The absolute bare minimum version of PEP 587 that we need for that
purpose is to expose the PreInitialize API, as that's the one that
allows the active text encoding to be set early enough to avoid
mojibake: https://www.python.org/dev/peps/pep-0587/#pre-initialization-with-pypreconfig

The rest of the proposal in PEP 587 then comes from wanting to publish
an API that matches the one we're now using ourselves, rather than
coming up with something more speculative.

However, I was also going to suggest that you (Steve) might make a
good BDFL-Delegate for these PEPs - there aren't that many of us
familiar with this part of the code base, and Victor, Eric, and I are
all way too close to the concrete API design to judge it objectively,
while you not only maintain the embeddable CPython bundle for Windows,
you also have access to users of that bundle that might be able to
provide you with additional feedback :)

Cheers,
Nick.

P.S. I've also posted a draft update to PEP 432 that modifies it to
reflect Victor's extraction of the part we already have as a private
API to PEP 587:https://github.com/python/peps/pull/965

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia