[Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version

[Victor Stinner]

Hi,

First of all, I just found an old issue that we will solved by my PEP 587 :-)

Add Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal errors
https://bugs.python.org/issue30560

I studied code of applications embedding Python. Most of them has to
decode bytes strings to get wchar_t* to set home, argv, program name,
etc. I'm not sure that they use the "correct" encoding, especially
since Python 3.7 got UTF-8 Mode (PEP 540) and C locale coercion (PEP
538).

I tried to convert the source code of each project into pseudo-code
which looks like C code used in CPython.

I removed all error handling code: look at each reference, the
original code is usually way more complex.

Some project has to wrap each function of the Python C API manually,
which adds even more boilerplate code.

Some project set/unset environment varaibles. Others prefer global
configuration variables like Py_NoSiteFlag.

It seems like Py_FrozenFlag is commonly used. Maybe I should make the
flag public and try to find it a better name:

    /* If greater than 0, suppress _PyPathConfig_Calculate() warnings.

       If set to -1 (default), inherit Py_FrozenFlag value. */
    int _frozen;

About pyinstaller which changes C standard stream buffering:
Py_Initialize() now also does that when buffered_stdio=0. See
config_init_stdio() in Python/coreconfig.c. Moreover, this function
now *always* set standard streams to O_BINARY mode on Windows. I'm not
sure if it's correct or not.


Blender
-------

Pseudo-code of BPY_python_start::

    BLI_strncpy_wchar_from_utf8(program_path_wchar, BKE_appdir_program_path());
    Py_SetProgramName(program_path_wchar);
    PyImport_ExtendInittab(bpy_internal_modules);
    Py_SetPythonHome(py_path_bundle_wchar);
    Py_SetStandardStreamEncoding("utf-8", "surrogateescape");
    Py_NoSiteFlag = 1;
    Py_FrozenFlag = 1;
    Py_Initialize();

Ref: https://git.blender.org/gitweb/gitweb.cgi/blender.git/blob/HEAD:/source/blender/python/intern/bpy_interface.c

fontforge
---------

Pseudo-code of fontforge when Python is used to run a script::

    Py_Initialize()
    for init_file in init_files:
       PyRun_SimpleFileEx(init_file)
    exitcode = Py_Main(arg, argv)
    Py_Finalize()
    exit(exitcode)

Ref: https://bugs.python.org/issue36204#msg337256

py2app
------

Pseudo-code::

    unsetenv("PYTHONOPTIMIZE");
    unsetenv("PYTHONDEBUG");
    unsetenv("PYTHONDONTWRITEBYTECODE");
    unsetenv("PYTHONIOENCODING");
    unsetenv("PYTHONDUMPREFS");
    unsetenv("PYTHONMALLOCSTATS");
    setenv("PYTHONDONTWRITEBYTECODE", "1", 1);
    setenv("PYTHONUNBUFFERED", "1", 1);
    setenv("PYTHONPATH", build_python_path(), 1);

    setlocale(LC_ALL, "en_US.UTF-8");
    mbstowcs(w_program, c_program, PATH_MAX+1);
    Py_SetProgramName(w_program);

    Py_Initialize()

    argv_new[0] = _Py_DecodeUTF8_surrogateescape(script, strlen(script));
    ...
    PySys_SetArgv(argc, argv_new);

    PyRun_SimpleFile(fp, script);
    Py_Finalize();

Ref: https://bitbucket.org/ronaldoussoren/py2app/src/default/py2app/apptemplate/src/main.c

See also: https://bitbucket.org/ronaldoussoren/py2app/src/default/py2app/bundletemplate/src/main.m


OpenOffice
----------

Pseudo-code of ``PythonInit``::

    mbstowcs(wide, home, PATH_MAX + 1);
    Py_SetPythonHome(wide);
    setenv("PYTHONPATH", getenv("PYTHONPATH") + ":" + path_bootstrap);
    PyImport_AppendInittab("pyuno", PyInit_pyuno);
    Py_DontWriteBytecodeFlag = 1;
    Py_Initialize();

Ref: pyuno/source/loader/pyuno_loader.cxx, see:
https://docs.libreoffice.org/pyuno/html/pyuno__loader_8cxx_source.html

vim
---

Pseudo-code::

    mbstowcs(py_home_buf, p_py3home);
    Py_SetPythonHome(py_home_buf);
    PyImport_AppendInittab("vim", Py3Init_vim);
    Py_Initialize();

Ref: https://github.com/vim/vim/blob/master/src/if_python3.c

pyinstaller
-----------

Pseudo-code::

    pyi_locale_char2wchar(progname_w, status->archivename)
    SetProgramName(progname_w);

    pyi_locale_char2wchar(pyhome_w, status->mainpath)
    SetPythonHome(pyhome_w);

    pypath_w = build_path();
    Py_SetPath(pypath_w);

    Py_NoSiteFlag = 1;
    Py_FrozenFlag = 1;
    Py_DontWriteBytecodeFlag = 1;
    Py_NoUserSiteDirectory = 1;
    Py_IgnoreEnvironmentFlag = 1;
    Py_VerboseFlag = 0;
    Py_OptimizeFlag = 1;

    if (unbuffered) {
    #ifdef _WIN32
        _setmode(fileno(stdin), _O_BINARY);
        _setmode(fileno(stdout), _O_BINARY);
    #endif
        setbuf(stdin, (char *)NULL);
        setbuf(stdout, (char *)NULL);
        setbuf(stderr, (char *)NULL);
    }

    Py_Initialize();

    PySys_SetPath(pypath_w);

    PySys_SetArgvEx(argc, wargv, 0);

Ref: https://github.com/pyinstaller/pyinstaller/blob/1844d69f5aa1d64d3feca912ed1698664a3faf3e/bootloader/src/pyi_pythonlib.c

Victor