On Mon, Sep 30, 2019 at 11:00 PM Brett Cannon
Victor Stinner wrote:
Hi Nick, Le dim. 29 sept. 2019 à 08:47, Nick Coghlan ncoghlan@gmail.com a écrit :
I don't quite understand the purpose of this change, as there's no stable ABI for applications embedding CPython. Well, I would like to prepare Python to provide a stable ABI for embedded Python. While it's not a design goal yet (Include/cpython/initconfig.h is currently excluded from Py_LIMITED_API), this change is a step towards that.
So then isn't this a very last-minute, premature optimization if it isn't a design goal yet? If that's the case then I would vote not to make the change and wait until there's been feedback on 3.8 and then look at stabilizing the embedding API such that it can be considered stable.
I just want to chime in here and confirm, as a Professional CPython Embedder(tm) that embedders cannot currently rely on a stable ABI, or the limited API, and often not even the public API. It's not just about exposed symbols and struct sizes, it's often also about the semantics of internals (e.g. how importlib handles custom module finders and loaders) that subtly changes. For anything but the simplest PyRun_SimpleString-based embedding this is more the case when embedding than extending, and even than a regular (complex) Python program. (I already showed Victor and a few others some of the hoops we have to jump through at Google to embed CPython correctly, and only half of those things are necessary just because of Google's environment.)
-Brett
As a result, updating to a new X.Y.0 release always requires rebuilding the entire application, not just building and relinking CPython. In Python 3.8, C extensions are no longer linked to libpython which allows to switch between a release build and a debug build of libpython. Can we imagine the same idea for embedded Python? I checked vim on Linux: it's linked to libpython3.7m.so.1.0: a specific Python version, library built in release mode. I could understand a change to require passing in an expected Python version so we can fail more gracefully on a bad link where an application that intended to embed Python 3.8 is incorrectly linked against Python 3.9 (for example), but performing that kind of check would require passing in PY_VERSION_HEX, not the size of the config struct. It seems simpler to me to pass the structure size rather than the Python version. It avoids the risk of updating the structure without update the Python version. I also avoids to have to change the Python version immediately when PyConfig is modified. The main risk of sizeof(PyConfig) comes if we remove a field and add a new field of the same size: the structure size doesn't change... But in my experience, we only add new ways to configure Pyhon, we never remove old ones :-D The question is if it's convenient to compute sizeof(PyConfig) in programming languages other than C. Providing a "structure version" or the structure size from a function call would not work. The value must be known a compilation time, not at runtime. The purpose is to compare the version/size between build and runtime (they must match). In the first implementation of my PEP, I used an internal "config version" provides by a macro. But it was said that macros are not convenient. PY_VERSION_HEX is provided as a macro, but we are now trying to avoid macros in our C API, no? At least, it's what I understood from the PEP 587 discussion. We don't support that - all our APIs that accept PyObject/PyTypeObject/etc require that the caller pass in structs of the correct size for the version of Python being used. For PyConfig, it's allocated (on the stack on or on heap) by the application. So the application requires to access to the full structure. Objects (instances) are allocated by Python (on the heap). Applications usually don't need to know/access the structure. Python is far from being perfect, static types are still supported and they are an issue for the stable ABI. The PyConfig and PyPreConfig structs are no different from PyObject in that regard: if there's a size mismatch, then the developers of the embedding application have somehow messed up their build process. In short, PyConfig initialization works like that:
The application allocates a PyConfig object on the stack Python calls memset(config, 0, sizeof(PyConfig))
If there is a size mismatch, Python triggers a buffer overflow which is likely to cause issues. I prefer to have a clean API which makes buffer overflow impossible. Embedding Python and handling different Python versions is not trivial, especially if we start to add new fields to each PyConfig (which is very likely). If prefer to be extra careful. I also expect bad surprises even in CPython with Programs/_testembed: many tests use PyConfig. Depending if _testembed is properly rebuilt or not, bad thing will happen. -- To implement my PEP 445 "Add new APIs to customize Python memory allocators", I added a PyMemAllocator structure which is part of the API. Quickly, I had to add a new field to PyMemAllocator. But it wasn't possible to detect if a C extension used the old or the new structure... So I decided to rename the structure to PyMemAllocatorEx to ensure that the compilation of all C extensions using the API will fail... :-( I really dislike this solution. What will happen when we will add another field to the structure, like a new PyMem_Aligned() (similar to posix_memalign()) function? PyMem_Aligned() can be implementation on top of an existing memory allocator which doesn't support it natively. But the problem is again the API and the PyMemAllocatorEx structure... Victor Night gathers, and now my watch begins. It shall not end until my death.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4ONRL3O6...
--
Thomas Wouters