Configure Python initialization (PyConfig) in Python
Hi, If you embed Python in Python, I would like your opinion on https://bugs.python.org/issue42260 issue which adds a way to configure the Python initialization in Python. I'm looking for feedback from people who embed Python and have issues to configure Python as they wish. Simple example (just to show how the API would look like) setting the the bytes_warning option: ---------------- import _testinternalcapi import sys assert sys.flags.bytes_warning == 0 config = _testinternalcapi.get_config() # Copy PyConfig config['bytes_warning'] = 1 # regular Python dict _testinternalcapi.set_config(config) # Update PyConfig and sys assert sys.flags.bytes_warning == 1 ---------------- The example already works in the current master branch! The PyConfig structure has 53 members (+ 4 private members). You can configure many things: sys.argv, sys.path, stdout encoding, disable user site directory, etc. In Python 3.8, I implemented PEP 587 "Python Initialization Configuration" which is the concrete implementation of the first half of PEP 432 "Restructuring the CPython startup sequence". Python now provides a C API to configure every option of its initialization: https://docs.python.org/dev/c-api/init_config.html Nick Coghlan recently withdrew his PEP 432 since PEP 587 was rejected. It made me sad :-( I tried to implement the second half of the PEP 432: * [DONE] Introduce the most limited "core" initialization phase where Python only has builtin types and builtin modules (implemented as a provisional C API in the PEP 587, I fixed some issues recently). sys module is only partially initialized (no sys.path, no sys.stdout). Python should not access the file system in his phase. * [DONE] Only compute the Python path configuration during the main initialization, not during the core initialization. * [DONE] Ability to configure the Python initialization in Python. * [WIP] Rewrite Modules/getpath.c (1600 lines of C code) to Lib/_getpath.py (700 lines of Python code) See https://bugs.python.org/issue42260 for the work-in-progress. In Python 3.9, it's already possible to configure PyConfig in C. The idea here is to give the ability to do it in Python, but also to reimplement the old Modules/getpath.c as modern Python code. In my WIP PR 23169, I reimplemented getpath.c in two parts: * _getpath Python module: logic to build sys.path * _cgetpath C extension used by _getpath: minimum C code to expose Py_DecodeLocale() function and some constants in Python It will be easier to maintain and evolve _getpath.py, but also to write a test suite (right now, getpath.c has almost no test, see test_embed). Later, the idea would be to let applications embedding Python to execute their own code to configure Python initialization options (PyConfig) in Python between the "core" initialization phase and the "main" initialization (Python fully initialized). It should not be run after the main initialization, since the configuration is used to configure Python during the main initialization. From a practical point of view, executing Python code to initialize Python requires freezing this code as a frozen module (as I did for _getpath.py in my PR). I don't know if it's easy to do that in an application embedding Python right now. My remaining problem is that my PR changes PyConfig_Read(): it no longer computes the Python Path Configuration, since it is now implemented in Python. IMO it's an acceptable drawback compared to the benefit of these new features. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Wed, 11 Nov 2020 at 09:59, Victor Stinner <vstinner@python.org> wrote:
Hi,
If you embed Python in Python, I would like your opinion on https://bugs.python.org/issue42260 issue which adds a way to configure the Python initialization in Python.
I'm looking for feedback from people who embed Python and have issues to configure Python as they wish.
Simple example (just to show how the API would look like) setting the the bytes_warning option: ---------------- import _testinternalcapi import sys
assert sys.flags.bytes_warning == 0
config = _testinternalcapi.get_config() # Copy PyConfig config['bytes_warning'] = 1 # regular Python dict _testinternalcapi.set_config(config) # Update PyConfig and sys
assert sys.flags.bytes_warning == 1 ----------------
The example already works in the current master branch!
Nice! I remember the work you put in getting runtime reconfiguration working again for 3.7.1, so it's cool to see that capability maturing even further.
Nick Coghlan recently withdrew his PEP 432 since PEP 587 was rejected.
s/rejected/accepted/ :)
It made me sad :-(
Seeing the huge improvements that have been made to the startup management code since I first wrote PEP 432 makes me happy. Sure, the technical details aren't exactly what I first envisioned, but I never expected those to survive exactly as written - the problem is simply too big and too complex for my first guess at a plausible way of fixing it to be exactly what we ended up doing :)
* [WIP] Rewrite Modules/getpath.c (1600 lines of C code) to Lib/_getpath.py (700 lines of Python code)
See https://bugs.python.org/issue42260 for the work-in-progress.
Wow, it's one thing to suggest this as a desirable goal, and another to see it so close to actually happening! Very cool :)
From a practical point of view, executing Python code to initialize Python requires freezing this code as a frozen module (as I did for _getpath.py in my PR). I don't know if it's easy to do that in an application embedding Python right now.
It's still not trivial, but it's easier now that we do it in the standard library (since it means there is reference code to copy).
My remaining problem is that my PR changes PyConfig_Read(): it no longer computes the Python Path Configuration, since it is now implemented in Python. IMO it's an acceptable drawback compared to the benefit of these new features.
Even though I wouldn't expect it to be controversial, it's probably worth writing up this migration of the path configuration code from C to Python as a PEP, along the lines of what we suggested in the last part of the PEP 432 withdrawal notice. The change has the potential to produce unexpected consequences for folks inadvertently relying on the quirks of the current implementation, and having the PEP there to cover the technical details means we can provide more information than could be squeezed into a single paragraph in the What's New document. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le sam. 14 nov. 2020 à 14:43, Nick Coghlan <ncoghlan@gmail.com> a écrit :
My remaining problem is that my PR changes PyConfig_Read(): it no longer computes the Python Path Configuration, since it is now implemented in Python. IMO it's an acceptable drawback compared to the benefit of these new features.
Even though I wouldn't expect it to be controversial, it's probably worth writing up this migration of the path configuration code from C to Python as a PEP, along the lines of what we suggested in the last part of the PEP 432 withdrawal notice.
The change has the potential to produce unexpected consequences for folks inadvertently relying on the quirks of the current implementation, and having the PEP there to cover the technical details means we can provide more information than could be squeezed into a single paragraph in the What's New document.
I agree that a PEP would be a good way to communicate the rationale and the API changes. I have to think about it ;-) The tricky part is always the backward compatibility part, since this change is clearly backward incompatible on purpose. Since it's a "new" API (Python 3.8, released in October 2019), maybe it would be acceptable. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
participants (2)
-
Nick Coghlan
-
Victor Stinner