[Python-Dev] Python initialization and embedded Python

[Nick Coghlan]

On 19 November 2017 at 01:45, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 18.11.17 16:17, Nick Coghlan пише:
>>
>> On 18 November 2017 at 10:01, Victor Stinner <victor.stinner at gmail.com>
>> wrote:
>>>
>>> I'm writing this email to ask if this change is an issue or not to
>>> embedded Python and the Python C API. Is it still possible to call
>>> "all" functions of the C API before calling Py_Initialize()?
>>
>>
>> It isn't technically permitted to call any of them, unless their
>> documentation specifically says that calling them before
>> `Py_Initialize` is permitted (and that permission is only given for a
>> select few configuration APIs in
>> https://docs.python.org/3/c-api/init.html).
>
> The Py_Initialize() is not complete. It mentions only Py_SetProgramName(),
> Py_SetPythonHome() and Py_SetPath(). But in other places it is documented
> that Py_SetStandardStreamEncoding(), PyImport_AppendInittab(),
> PyImport_ExtendInittab() should be called before Py_Initialize(). And the
> embedding examples call Py_DecodeLocale() before Py_Initialize().
> PyMem_RawMalloc(), PyMem_RawFree() and PyInitFrozenExtensions() are called
> before Py_Initialize() in Py_FrozenMain(). Also these functions call
> _PyMem_RawStrdup().
>
> Hence, the minimal set of functions that can be called before
> Py_Initialize() is:
>
> * Py_SetProgramName()
> * Py_SetPythonHome()
> * Py_SetPath()
> * Py_SetStandardStreamEncoding()
> * PyImport_AppendInittab()
> * PyImport_ExtendInittab()
> * Py_DecodeLocale()
> * PyMem_RawMalloc()
> * PyMem_RawFree()
> * PyInitFrozenExtensions()

OK, in that case I think the answer to Victor's question is:

1. Breaking calling Py_DecodeLocale() before calling Py_Initialize()
is a compatibility break with the API implied by our own usage
examples, and we'll need to revert the breakage for 3.7, and ensure at
least one release's worth of DeprecationWarning before requiring
either the use of an alternative API (where the caller controls the
memory management), or else a new lower level pre-initialization API
(i.e. making `PyRuntime_Initialize` a public API)
2. We should provide a consolidated list of these functions in the C
API initialization docs
3. We should add more test cases to _testembed.c that ensure they all
work correctly prior to Py_Initialize (some of them are already tested
there, but definitely not all of them)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia