[Python-Dev] Python initialization and embedded Python

[Nick Coghlan]

On 24 November 2017 at 09:19, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> We are close to the 3.7a3 release and the bug is not fixed yet. I
> propose to revert the changes on memory allocators right now, and take
> time to design a proper fix which will respect all constraints.
>
> https://github.com/python/cpython/pull/4532
>
> Today, someone came to me on IRC to complain that calling
> Py_DecodeLocale() does now crash on Python 3.7. He is doing tests to
> embed Python on Android. Later he asks me about
> PyImport_AppendInittab(), but I don't know this function. He told me
> that it does crash in PyMem_Realloc()... But PyImport_AppendInittab()
> must be called before Py_Initialize()...
>
> It confirms that Python is embedded and that the C API is used before
> Py_Initialize().
>
> We don't know yet exactly how the the C API is used, which functions
> are called before Py_Initialize().


We do note some of them explicitly at
https://docs.python.org/3/c-api/init.html (search for "before Py").

What we've been missing is a test case that ensures
https://docs.python.org/3/extending/embedding.html#very-high-level-embedding
actually works reliably (hence how we managed to break it by way of the
internal state management refactoring).

Once that core regression has been fixed, we can review the docs and the
test suite and come up with:

- a consolidated list of *all* the APIs that can safely be called before
Py_Initialize
- one or more new or updated test cases to ensure that any not yet tested
pre-initialization APIs actually work as intended


> Moreover, PEP 432 implementation is
> still incomplete, and calling _PyRuntime_Initialize() is just not
> possible, since it's a private API which is not exported...
>

Even after we reach the point of exposing the more fine-grained
initialisation API (which I'm now thinking we may be able to do for 3.8
given Eric & Victor's work on it for 3.7), we're still going to have to
ensure the existing configuration API keeps working as expected.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171124/f4cd5f0f/attachment.html>