PEP: Hide implementation details in the C API
Hi, This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed. For the background, see also my talk at the previous Python Language Summit at Pycon US, Portland OR: "Keeping Python competitive" https://lwn.net/Articles/723752/#723949 "Python performance", slides (PDF): https://github.com/haypo/conf/raw/master/2017-PyconUS/summit.pdf Since this is really the first draft, I didn't assign a PEP number to it yet. I prefer to wait for a first feedback round. Victor PEP: xxx Title: Hide implementation details in the C API Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner@gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 31-May-2017 Abstract ======== Modify the C API to remove implementation details. Add an opt-in option to compile C extensions to get the old full API with implementation details. The modified C API allows to more easily experiment new optimizations: * Indirect Reference Counting * Remove Reference Counting, New Garbage Collector * Remove the GIL * Tagged pointers Reference counting may be emulated in a future implementation for backward compatibility. Rationale ========= History of CPython forks ------------------------ Last 10 years, CPython was forked multiple times to attempt different CPython enhancements: * Unladen Swallow: add a JIT compiler based on LLVM * Pyston: add a JIT compiler based on LLVM (CPython 2.7 fork) * Pyjion: add a JIT compiler based on Microsoft CLR * Gilectomy: remove the Global Interpreter Lock nicknamed "GIL" * etc. Sadly, none is this project has been merged back into CPython. Unladen Swallow looses its funding from Google, Pyston looses its funding from Dropbox, Pyjion is developed in the limited spare time of two Microsoft employees. One hard technically issue which blocked these projects to really unleash their power is the C API of CPython. Many old technical choices of CPython are hardcoded in this API: * reference counting * garbage collector * C structures like PyObject which contains headers for reference counting and the garbage collector * specific memory allocators * etc. PyPy ---- PyPy uses more efficient structures and use a more efficient garbage collector without reference counting. Thanks to that (but also many other optimizations), PyPy succeeded to run Python code up to 5x faster than CPython. Plan made of multiple small steps ================================= Step 1: split Include/ into subdirectories ------------------------------------------ Split the ``Include/`` directory of CPython: * ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake. At this point, ``Include/Python.h`` is not changed at all: zero risk of backward incompatibility. The ``core`` API is the most complete API exposing *all* implementation details and use macros for best performances. XXX should we abandon the stable ABI? Never really used by anyone. Step 2: Add an opt-in API option to tools building packages ----------------------------------------------------------- Modify Python packaging tools (distutils, setuptools, flit, pip, etc.) to add an opt-in option to choose the API: ``python``, ``core`` or ``stable``. For example, debuggers like ``vmprof`` need need the ``core`` API to get a full access to implementation details. XXX handle backward compatibility for packaging tools. Step 3: first pass of implementation detail removal --------------------------------------------------- Modify the ``python`` API: * Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory. * Slowly remove more and more implementation details from this API. Modifications of these API should be driven by tests of popular third party packages like: * Django with database drivers * numpy * scipy * Pillow * lxml * etc. Compilation errors on these extensions are expected. This step should help to draw a line for the backward incompatible change. Goal: remove a few implementation details but don't break numpy and lxml. Step 4 ------ Switch the default API to the new restricted ``python`` API. Help third party projects to patch their code: don't break the "Python world". Step 5 ------ Continue Step 3: remove even more implementation details. Long-term goal to complete this PEP: Remove *all* implementation details, remove all structures and macros. Alternative: keep core as the default API ========================================= A smoother transition would be to not touch the existing API but work on a new API which would only be used as an opt-in option. Similar plan used by Gilectomy: opt-in option to get best performances. It would be at least two Python binaries per Python version: default compatible version, and a new faster but incompatible version. Idea: implementation of the C API supporting old Python versions? ================================================================= Open questions. Q: Would it be possible to design an external library which would work on Python 2.7, Python 3.4-3.6, and the future Python 3.7? Q: Should such library be linked to libpythonX.Y? Or even to a pythonX.Y binary which wasn't built with shared library? Q: Would it be easy to use it? How would it be downloaded and installed to build extensions? Collaboration with PyPy, IronPython, Jython and MicroPython =========================================================== XXX to be done Enhancements becoming possible thanks to a new C API ==================================================== Indirect Reference Counting --------------------------- * Replace ``Py_ssize_t ob_refcnt;`` (integer) with ``Py_ssize_t *ob_refcnt;`` (pointer to an integer). * Same change for GC headers? * Store all reference counters in a separated memory block (or maybe multiple memory blocks) Expected advantage: smaller memory footprint when using fork() on UNIX which is implemented with Copy-On-Write on physical memory pages. See also `Dismissing Python Garbage Collection at Instagram <https://engineering.instagram.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172>`_. Remove Reference Counting, New Garbage Collector ------------------------------------------------ If the new C API hides well all implementation details, it becomes possible to change fundamental features like how CPython tracks the lifetime of an object. * Remove ``Py_ssize_t ob_refcnt;`` from the PyObject structure * Replace the current XXX garbage collector with a new tracing garbage collector * Use new macros to define a variable storing an object and to set the value of an object * Reimplement Py_INCREF() and Py_DECREF() on top of that using a hash table: object => reference counter. XXX PyPy is only partially successful on that project, cpyext remains very slow. XXX Would it require an opt-in option to really limit backward compatibility? Remove the GIL -------------- * Don't remove the GIL, but replace the GIL with smaller locks * Builtin mutable types: list, set, dict * Modules * Classes * etc. Backward compatibility: * Keep the GIL Tagged pointers --------------- https://en.wikipedia.org/wiki/Tagged_pointer Common optimization, especially used for "small integers". Current C API doesn't allow to implement tagged pointers. Tagged pointers are used in MicroPython to reduce the memory footprint. Note: ARM64 was recently extended its address space to 48 bits, causing issue in LuaJIT: `47 bit address space restriction on ARM64 <https://github.com/LuaJIT/LuaJIT/issues/49>`_. Misc ideas ---------- * Software Transactional Memory? See `PyPy STM <http://doc.pypy.org/en/latest/stm.html>`_ Idea: Multiple Python binaries ============================== Instead of a single ``python3.7``, providing two or more binaries, as PyPy does, would allow to experiment more easily changes without breaking the backward compatibility. For example, ``python3.7`` would remain the default binary with reference counting and the current garbage collector, whereas ``fastpython3.7`` would not use reference counting and a new garbage collector. It would allow to more quickly "break the backward compatibility" and make it even more explicit than only prepared C extensions will be compatible with the new ``fastpython3.7``. cffi ==== XXX Long term goal: "more cffi, less libpython". Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On 11 July 2017 at 20:19, Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed.
Please don't use the word "needed" for speed increases, as we can just as well diagnose the problem with the status quo as "The transition from writing and publishing pure Python modules to writing and publishing pre-compiled accelerated modules in Cython, C, C++, Rust, or Go is too hard, so folks mistakenly think they need to rewrite their whole application in something else, rather than just selectively replacing key pieces of it". After all, we already *have* an example of a breakout success for writing Python applications that rival C and FORTRAN applications for raw speed: Cython. By contrast, most of the alternatives that have attempted to make Python faster without forcing users to give up on some the language's dynamism in the process have been plagued by compatibility challenges and found themselves needing to ask for the language's runtime semantics to be relaxed in one way or another. That said, trying to improve how we manage the distinction between the public API and the interpreter's internal APIs is still an admirable goal, and it would be *great* to have CPython natively provide the public API that cffi relies on (so that other projects could also effectively target it), so my main request is just to reign in the dramatic rhetoric and start by exploring how many of the benefits you'd like can be obtained *without* a hard compatibility break. While the broad C API is one of CPython's greatest strengths that enabled the Python ecosystem to become the powerhouse that it is, it is *also* a pain to maintain consistently, *and* it poses problems for some technical experiments various folks would like to carry out. Those kinds of use cases are more than enough to justify changes to the way we manage our public header files - you don't need to dress it up in "sky is falling" rhetoric founded in the fear of other programming languages. Yes, Python is a nice language to program in, and it's great that we can get jobs where we can get paid to program in it. That doesn't mean we have to treat it as an existential threat that we aren't always going to be the best choice for everything :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 11 July 2017 at 11:19, Victor Stinner <victor.stinner@gmail.com> wrote:
XXX should we abandon the stable ABI? Never really used by anyone.
Please don't. On Windows, embedding Python is a pain because a new version of Python requires a recompile (which isn't ideal for apps that just want to optionally allow Python scripting, for example). Also, the recent availability of the embedded distribution on Windows has opened up some opportunities and I've been using the stable ABI there. It's not the end of the world if we lose it, but I'd rather see it retained (or better still, enhanced). Paul
Step 3: first pass of implementation detail removal ---------------------------------------------------
Modify the ``python`` API:
* Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory. * Slowly remove more and more implementation details from this API.
When I discussed this issue with Serhiy Storchaka, he didn't see the purpose of the API directory. I started to implement the PEP in my "capi2" fork of CPython: https://github.com/haypo/cpython/tree/capi2 See https://github.com/haypo/cpython/tree/capi2/API for examples of C code to "implement the C API". Just one example, the macro #define PyUnicode_IS_READY(op) (((PyASCIIObject*)op)->state.ready) is replaced with a function: int PyUnicode_IS_READY(const PyObject *op) { return ((PyASCIIObject*)op)->state.ready; } So the header file doesn't have to expose the PyASCIIObject, PyCompactUnicodeObject and PyUnicodeObject structures. I was already able to remove the PyUnicodeObject structure without breaking the C extensions of the stdib. I don't want to pollute Objects/unicodeobject.c with such "wrapper" functions. In the future, the implementation of API/ can evolve a lot. Victor
On Tue, Jul 11, 2017 at 4:19 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
Step 1: split Include/ into subdirectories ------------------------------------------
Split the ``Include/`` directory of CPython:
* ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI
Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake.
At this point, ``Include/Python.h`` is not changed at all: zero risk of backward incompatibility.
The ``core`` API is the most complete API exposing *all* implementation details and use macros for best performances.
FWIW, this is similar to something I've done while working on gathering up the CPython global runtime state. [1] I needed to share some internal details across compilation modules. Nick suggested a separate Includes/internal directory for header files containing "private" API. There is a _Python.h file there that starts with: #ifndef Py_BUILD_CORE #error "Internal headers are not available externally." #endif In Includes/Python.h, Includes/internal/_Python.h gets included if Py_BUILD_CORE is defined. This approach makes a strict boundary that keeps internal details out of the public API. That way we don't accidentally leak private API. It sounds similar to this part of your proposal (adding the "core" API). -eric [1] http://bugs.python.org/issue30860
This is a great idea. The suggestions in your first draft would help clean up some of the uglier corners of the PyCXX code. I'd suggest that you might want to add at least 1 PyCXX based extension to your testing. PyCXX aims to expose all the C API as C++ classes. (I'm missing the class variable support as you discussed on this list a while ago). I'd be happy to support your API in PyCXX. Barry
Commenting more on specific technical details rather than just tone this time :) On 11 July 2017 at 20:19, Victor Stinner <victor.stinner@gmail.com> wrote:
PEP: xxx Title: Hide implementation details in the C API Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner@gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 31-May-2017
Abstract ========
Modify the C API to remove implementation details. Add an opt-in option to compile C extensions to get the old full API with implementation details.
The modified C API allows to more easily experiment new optimizations:
* Indirect Reference Counting * Remove Reference Counting, New Garbage Collector * Remove the GIL * Tagged pointers
Reference counting may be emulated in a future implementation for backward compatibility.
I don't believe this is the best rationale to use for the PEP, as we (or at least I) have emphatically promised *not* to do another Python 3 style compatibility break, and we know from PyPy's decade of challenges that a lot of Python's users care even more about CPython C API/ABI compatibility than they do the core data model. It also has the downside of not really being true, since *other implementations* are happily experimenting with alternative approaches, and projects like PyMetabiosis attempt to use CPython itself as an adapter between other runtimes and the full C API for those extension modules that need it. What is unequivocally true though is that in the current C API: 1. We're not sure which APIs other projects (including extension module generators and helper libraries like Cython, Boost, PyCXX, SWIG, cffi, etc) are *actually* relying on. 2. It's easy for us to accidentally expand the public C API without thinking about it, since Py_BUILD_CORE guards are opt-in and Py_LIMITED_API guards are opt-out 3. We haven't structured our header files in a way that makes it obvious at a glance which API we're modifying (internal API, public API, stable ABI)
Rationale =========
History of CPython forks ------------------------
Last 10 years, CPython was forked multiple times to attempt different CPython enhancements:
* Unladen Swallow: add a JIT compiler based on LLVM * Pyston: add a JIT compiler based on LLVM (CPython 2.7 fork) * Pyjion: add a JIT compiler based on Microsoft CLR * Gilectomy: remove the Global Interpreter Lock nicknamed "GIL" * etc.
Sadly, none is this project has been merged back into CPython. Unladen Swallow looses its funding from Google, Pyston looses its funding from Dropbox, Pyjion is developed in the limited spare time of two Microsoft employees.
One hard technically issue which blocked these projects to really unleash their power is the C API of CPython.
This is a somewhat misleadingly one-sided presentation of Python's history, as the broad access to CPython internals offered by the C API is precisely what *enabled* the scientific Python stack (including NumPy, SciPy, Pandas, scikit-learn, Cython, Numba, PyCUDA, etc) to develop largely independently of CPython itself. So for folks that are willing to embrace the use of Cython (and extension modules in general), many of CPython's runtime limitations (like the GIL and the overheads of working with boxed values) can already be avoided by pushing particular sections of code closer to C semantics than they are to traditional Python semantics. We've also been working to bring the runtime semantics of extension modules ever closer to those of pure Python modules, to the point where Python 3.7 is likely to be able to run an extension module as __main__ (see https://www.python.org/dev/peps/pep-0547/ for details)
Many old technical choices of CPython are hardcoded in this API:
* reference counting * garbage collector * C structures like PyObject which contains headers for reference counting and the garbage collector * specific memory allocators * etc.
PyPy ----
PyPy uses more efficient structures and use a more efficient garbage collector without reference counting. Thanks to that (but also many other optimizations), PyPy succeeded to run Python code up to 5x faster than CPython.
This framing makes it look a bit like you're saying "It's hard for PyPy to correctly emulate these aspects of CPython, so we should eliminate them as a barrier to adoption for PyPy by breaking them for currently happy CPython's users as well". I don't think that's really a framing you want to run with in the near term, as it's going to start a needless fight, when there's plenty of unambiguously beneficial work that coule be done before anyone starts contemplating any kind of API compatibility break :) In particular, better segmenting our APIs into "solely for CPython's internal use", "ABI is specific to a CPython version", "API is portable across Python implementations", "ABI is portable across CPython versions (and maybe even Python implementations)" allows tooling developers and extension module authors to make more informed decisions about how closely they want to couple their work to CPython specifically. And then *after* we've done that API clarification work, *then* we can ask the question about what the default behaviour of "#include <Python.h>" should be, and perhaps introduce an opt-in Py_CPYTHON_API flag to request access to the full traditional C API for extension modules and embedding applications that actually need it. (While that's still a compatibility break, it's one that can be trivially resolved by putting an unconditional "#define Py_CPYTHON_API" before the Python header inclusion for projects that find they were actually relying on CPython specifics)
Plan made of multiple small steps =================================
Step 1: split Include/ into subdirectories ------------------------------------------
Split the ``Include/`` directory of CPython:
* ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI
Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake.
At this point, ``Include/Python.h`` is not changed at all: zero risk of backward incompatibility.
The ``core`` API is the most complete API exposing *all* implementation details and use macros for best performances.
This part I like, although as Eric noted, we can avoid making wholesale changes to the headers of our implementation files by putting a Py_BUILD_CORE guard around the inclusion of a "Include/core/_CPython.h" header from "Include/Python.h"
XXX should we abandon the stable ABI? Never really used by anyone.
It's also not available in Python 2.7, so anyone straddling the 2/3 boundary isn't currently able to rely on it. As folks become more willing to drop Python 2.7 support, then expending the effort to start targeting the stable ABI instead becomes more attractive (especially for extension module creation tools like Cython, cffi, and SWIG), since the stable ABI usage can *replace* the code that uses the traditional CPython API.
Step 2: Add an opt-in API option to tools building packages -----------------------------------------------------------
Modify Python packaging tools (distutils, setuptools, flit, pip, etc.) to add an opt-in option to choose the API: ``python``, ``core`` or ``stable``.
For example, debuggers like ``vmprof`` need need the ``core`` API to get a full access to implementation details.
XXX handle backward compatibility for packaging tools.
For handcoded extensions, defining which API to use would be part of the C/C++ code. For generated extensions, it would be an option passed to Cython, cffi, etc. Packaging frontends shouldn't need to explicitly support it any more than they explicitly support the stable ABI today.
Step 3: first pass of implementation detail removal ---------------------------------------------------
Modify the ``python`` API:
* Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory. * Slowly remove more and more implementation details from this API.
I'd suggest doing this slightly differently by ensuring that the APIs are defined as strict supersets of each other as follows: 1. CPython internal APIs (Py_BUILD_CORE) 2. CPython C API (status quo, currently no qualifier) 3. Portable Python API (new, starts as equivalent to stable ABI) 4. Stable Python ABI (Py_LIMITED_API) The two new qualifiers would then be: #define Py_CPYTHON_API #define Py_PORTABLE_API And Include/Python.h would end up looking something like this: [Common configuration includes would still go here] #ifdef Py_BUILD_CORE #include "core/_CPython.h" #else #ifdef Py_LIMITED_API #include "stable/Python.h" #else #ifdef Py_PORTABLE_API #include "portable/Python.h" #else #define Py_CPYTHON_API #include "cpython/Python.h" #endif #endif #endif At some future date, the default could then potentially switch to being the portable API for the current Python version, with folks having to opt-in to using either the full CPython API or the portable API for an older version. To avoid having to duplicate prototype definitions, and to ensure that C compilers complain when we inadvertently redefine a symbol differently from the way a more restricted API defines it, each API superset would start by including the next narrower API. So we'd have this: Include/stable/Python.h: [No special preamble, as it's the lowest common denominator API] Include/portable/Python.h: #define Py_LIMITED_API Py_PORTABLE_API #include "../stable/Python.h" #undef Py_LIMITED_API [Any desired API additions and overrides] Include/cpython/Python.h: #include "../patchlevel.h" #define Py_PORTABLE_API PY_VERSION_HEX #include "../portable/Python.h" #undef Py_PORTABLE_API [Include the rest of the current public C API] Include/core/_CPython.h: #ifndef Py_BUILD_CORE #error "Internal headers are only available when building CPython" #endif #include "../cpython/Python.h" [Include the rest of the internal C API] And at least initially, the subdirectories would be mostly empty - instead, we'd have the following setup: 1. Unported headers would remain directly in "Include/" and be included from "Include/Python.h" 2. Ported headers would have their contents split between core, cpython, and stable based on their #ifdef chains 3. When porting, the more expansive APIs would use "#undef" as needed when overriding a symbol deliberately And then, once all the APIs had been clearly categorised in a way that C compilers can better help us manage, the folks that were interested in this could start building key extension modules (such as NumPy and lxml) using "Py_PORTABLE_API=0x03070000", and *adding* to the portable API on an explicitly needs-driven basis. Cheers, NIck. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7/11/2017 11:30 PM, Nick Coghlan wrote:
Commenting more on specific technical details rather than just tone this time :)
On 11 July 2017 at 20:19, Victor Stinner <victor.stinner@gmail.com> wrote:
Reference counting may be emulated in a future implementation for backward compatibility.
One heavy user's experience with garbage collection: https://engineering.instagram.com/dismissing-python-garbage-collection-at-in... "Instagram can run 10% more efficiently. ... Yes, you heard it right! By disabling GC [and relying only on ref counting], we can reduce the memory footprint and improve the CPU LLC cache hit ratio" It turned out that gc.disable() was inadequate because imported libraries could turn it on, and one did.
I don't believe this is the best rationale to use for the PEP, as we (or at least I) have emphatically promised *not* to do another Python 3 style compatibility break, and we know from PyPy's decade of challenges that a lot of Python's users care even more about CPython C API/ABI compatibility than they do the core data model.
[snip most] -- Terry Jan Reedy
On 11 Jul 2017, at 12:19, Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed.
I’m not sure if hiding implementation details will help a lot w.r.t. making CPython more efficient, but cleaning up the public API would avoid accidentally depending on non-public information (and is sound engineering anyway). That said, a lot of care should be taken to avoid breaking existing extensions as the ease of writing extensions is one of the strong points of CPython.
Plan made of multiple small steps =================================
Step 1: split Include/ into subdirectories ------------------------------------------
Split the ``Include/`` directory of CPython:
* ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI
Looks good in principle. It is currently too easy to accidentally add to the stable ABI by forgetting to add ‘#if’ guards around a non-stable API.
Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake.
Not sure about this, shouldn’t it be possible to have ``python`` include ``core`` and ``core`` include ``stable``? This would avoid having to update multiple header files when adding new definitions.
At this point, ``Include/Python.h`` is not changed at all: zero risk of backward incompatibility.
The ``core`` API is the most complete API exposing *all* implementation details and use macros for best performances.
XXX should we abandon the stable ABI? Never really used by anyone.
Assuming that’s true, has anyone looked into why it is barely used? If I’d have to guess its due to inertia.
Step 3: first pass of implementation detail removal ---------------------------------------------------
Modify the ``python`` API:
* Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory.
In this particular case (Py_INCREF/DECREF) making them functions isn’t really useful and is likely to be harmful for performance. It is not useful because these macros manipulate state in a struct that must be public because that struct is included into the structs for custom objects (PyObject_HEAD). Having them as macro’s also doesn’t preclude moving to indirect reference counts. Moving to anything that isn’t reference counts likely needs changes to the API (but not necessarily, see PyPy’s cpext).
* Slowly remove more and more implementation details from this API.
Modifications of these API should be driven by tests of popular third party packages like:
* Django with database drivers * numpy * scipy * Pillow * lxml * etc.
Compilation errors on these extensions are expected. This step should help to draw a line for the backward incompatible change.
This could also help to find places where the documented API is not sufficient. One of the places where I poke directly into implementation details is a C-level subclass of str (PyUnicode_Type). I’d prefer not doing that, but AFAIK there is no other way to be string-like to the C API other than by being a subclass of str. BTW. The reason I need to subclass str: in PyObjC I use a subclass of str to represent Objective-C strings (NSString/NSMutableString), and I need to keep track of the original value; mostly because there are some Objective-C APIs that use object identity. The worst part is that fully initialising the PyUnicodeObject fields often isn’t necessary as a lot of Objective-C strings aren’t used as strings in Python code.
Enhancements becoming possible thanks to a new C API ====================================================
Indirect Reference Counting ---------------------------
* Replace ``Py_ssize_t ob_refcnt;`` (integer) with ``Py_ssize_t *ob_refcnt;`` (pointer to an integer). * Same change for GC headers? * Store all reference counters in a separated memory block (or maybe multiple memory blocks)
This could be done right now with a minimal change to the API: just make the ob_refcnt and ob_type fields of the PyObject struct private by renaming them, in Py3 the documented way to access theses fields is through function macros and these could by changed to do indirect refcounting instead.
Tagged pointers ---------------
https://en.wikipedia.org/wiki/Tagged_pointer
Common optimization, especially used for "small integers".
Current C API doesn't allow to implement tagged pointers.
Why not? Thanks to Py_TYPE and Py_INCREF/Py_DECREF it should be possible to use tagged pointers without major changes to the API (also: see above).
Tagged pointers are used in MicroPython to reduce the memory footprint.
Note: ARM64 was recently extended its address space to 48 bits, causing issue in LuaJIT: `47 bit address space restriction on ARM64 <https://github.com/LuaJIT/LuaJIT/issues/49>`_.
That shouldn’t be a problem when only using the least significant bits as tag bits (those bits that are known to be zero in untagged pointers due to alignment).
Idea: Multiple Python binaries ==============================
Instead of a single ``python3.7``, providing two or more binaries, as PyPy does, would allow to experiment more easily changes without breaking the backward compatibility.
For example, ``python3.7`` would remain the default binary with reference counting and the current garbage collector, whereas ``fastpython3.7`` would not use reference counting and a new garbage collector.
It would allow to more quickly "break the backward compatibility" and make it even more explicit than only prepared C extensions will be compatible with the new ``fastpython3.7``.
The cost is having to maintain both indefinitely. Ronald
On Wed, 12 Jul 2017 at 01:25 Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 11 Jul 2017, at 12:19, Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed.
I’m not sure if hiding implementation details will help a lot w.r.t. making CPython more efficient, but cleaning up the public API would avoid accidentally depending on non-public information (and is sound engineering anyway). That said, a lot of care should be taken to avoid breaking existing extensions as the ease of writing extensions is one of the strong points of CPython.
I also think the motivation doesn't have to be performance but simply cleaning up how we expose our C APIs to users as shown by the fact we have messed up the stable API by making it opt-out instead of opt-in.
Plan made of multiple small steps =================================
Step 1: split Include/ into subdirectories ------------------------------------------
Split the ``Include/`` directory of CPython:
* ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI
Looks good in principle. It is currently too easy to accidentally add to the stable ABI by forgetting to add ‘#if’ guards around a non-stable API.
Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake.
Not sure about this, shouldn’t it be possible to have ``python`` include ``core`` and ``core`` include ``stable``? This would avoid having to update multiple header files when adding new definitions.
Yeah, that's also what I initially thought. Use a cascading hierarchy so that people know they should put anything as high up as possible to minimize its exposure. [SNIP]
Step 3: first pass of implementation detail removal ---------------------------------------------------
Modify the ``python`` API:
* Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory.
In this particular case (Py_INCREF/DECREF) making them functions isn’t really useful and is likely to be harmful for performance. It is not useful because these macros manipulate state in a struct that must be public because that struct is included into the structs for custom objects (PyObject_HEAD). Having them as macro’s also doesn’t preclude moving to indirect reference counts. Moving to anything that isn’t reference counts likely needs changes to the API (but not necessarily, see PyPy’s cpext).
I think Victor has long-term plans to try and hide the struct details at a higher-level and so that would make macros a bad thing. But ignoring the specific Py_INCREF/DECREF example, switching to functions does buy us the ability to actually change the function implementations between Python versions compared to having to worry about what a macro used to do (which is a possibility with the stable ABI).
* Slowly remove more and more implementation details from this API.
Modifications of these API should be driven by tests of popular third party packages like:
* Django with database drivers * numpy * scipy * Pillow * lxml * etc.
Compilation errors on these extensions are expected. This step should help to draw a line for the backward incompatible change.
This could also help to find places where the documented API is not sufficient. One of the places where I poke directly into implementation details is a C-level subclass of str (PyUnicode_Type). I’d prefer not doing that, but AFAIK there is no other way to be string-like to the C API other than by being a subclass of str.
Yeah, this would allow us to very clearly know what should or should not be documented (I would say the same for the stdlib but we all know old code didn't hide things with a leading underscore consistently).
BTW. The reason I need to subclass str: in PyObjC I use a subclass of str to represent Objective-C strings (NSString/NSMutableString), and I need to keep track of the original value; mostly because there are some Objective-C APIs that use object identity. The worst part is that fully initialising the PyUnicodeObject fields often isn’t necessary as a lot of Objective-C strings aren’t used as strings in Python code.
Enhancements becoming possible thanks to a new C API ====================================================
Indirect Reference Counting ---------------------------
* Replace ``Py_ssize_t ob_refcnt;`` (integer) with ``Py_ssize_t *ob_refcnt;`` (pointer to an integer). * Same change for GC headers? * Store all reference counters in a separated memory block (or maybe multiple memory blocks)
This could be done right now with a minimal change to the API: just make the ob_refcnt and ob_type fields of the PyObject struct private by renaming them, in Py3 the documented way to access theses fields is through function macros and these could by changed to do indirect refcounting instead.
I think this is why Victor wants functions, because even if you change the names the macros will be locked into their implementations if you try to write code that supports multiple versions and so you can't change it per-version of Python. -Brett
2017-07-12 20:51 GMT+02:00 Brett Cannon <brett@python.org>:
I also think the motivation doesn't have to be performance but simply cleaning up how we expose our C APIs to users as shown by the fact we have messed up the stable API by making it opt-out instead of opt-in.
It's hard to sell a "cleanup" to users with no carrot :-) Did someone remind trying to sell the "Python 3 cleanup"? :-)
Yeah, that's also what I initially thought. Use a cascading hierarchy so that people know they should put anything as high up as possible to minimize its exposure.
Yeah, maybe we can do that. I have to make my own experiment to make sure that #include doesn't leak symbols by mistakes and that it's still possible to use optimized macros or functions in builtin modules.
I think Victor has long-term plans to try and hide the struct details at a higher-level and so that would make macros a bad thing. But ignoring the specific Py_INCREF/DECREF example, switching to functions does buy us the ability to actually change the function implementations between Python versions compared to having to worry about what a macro used to do (which is a possibility with the stable ABI).
I think that my PEP is currently badly written :-) In fact, the idea is just to make the stable ABI usable :-) Instead of hiding structures *and* remove macros, my idea is just to hide structures but still provides macros... as functions. Basically, it will be the same API, but usable on more various implementations of Python. Victor
On 13 July 2017 at 21:46, Victor Stinner <victor.stinner@gmail.com> wrote:
2017-07-12 20:51 GMT+02:00 Brett Cannon <brett@python.org>:
I think Victor has long-term plans to try and hide the struct details at a higher-level and so that would make macros a bad thing. But ignoring the specific Py_INCREF/DECREF example, switching to functions does buy us the ability to actually change the function implementations between Python versions compared to having to worry about what a macro used to do (which is a possibility with the stable ABI).
I think that my PEP is currently badly written :-)
In fact, the idea is just to make the stable ABI usable :-) Instead of hiding structures *and* remove macros, my idea is just to hide structures but still provides macros... as functions. Basically, it will be the same API, but usable on more various implementations of Python.
As far as I know, this isn't really why folks find the stable ABI hard to switch to. Rather, I believe it's because switching to the stable ABI means completely changing how you define classes to be closer to the way you define them from Python code. That's why I like the idea of defining a "portable" API that *doesn't* adhere to the "no public structs" rule - if we can restore support for static class declarations (which requires exposing all the static method structs as well as the object header structs, although perhaps with obfuscated field names to avoid any dependency on the details of CPython's reference counting model), I think such an API would have dramatically lower barriers to adoption than the stable ABI does. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2017-07-13 15:21 GMT+02:00 Nick Coghlan <ncoghlan@gmail.com>:
As far as I know, this isn't really why folks find the stable ABI hard to switch to. Rather, I believe it's because switching to the stable ABI means completely changing how you define classes to be closer to the way you define them from Python code.
That's why I like the idea of defining a "portable" API that *doesn't* adhere to the "no public structs" rule - if we can restore support for static class declarations (which requires exposing all the static method structs as well as the object header structs, although perhaps with obfuscated field names to avoid any dependency on the details of CPython's reference counting model), I think such an API would have dramatically lower barriers to adoption than the stable ABI does.
I am not aware of this issue. Can you give an example of missing feature in the stable ABI? Or maybe an example of a class definition in C which cannot be implemented with the stable ABI? Victor
On 14 July 2017 at 01:35, Victor Stinner <victor.stinner@gmail.com> wrote:
2017-07-13 15:21 GMT+02:00 Nick Coghlan <ncoghlan@gmail.com>:
As far as I know, this isn't really why folks find the stable ABI hard to switch to. Rather, I believe it's because switching to the stable ABI means completely changing how you define classes to be closer to the way you define them from Python code.
That's why I like the idea of defining a "portable" API that *doesn't* adhere to the "no public structs" rule - if we can restore support for static class declarations (which requires exposing all the static method structs as well as the object header structs, although perhaps with obfuscated field names to avoid any dependency on the details of CPython's reference counting model), I think such an API would have dramatically lower barriers to adoption than the stable ABI does.
I am not aware of this issue. Can you give an example of missing feature in the stable ABI? Or maybe an example of a class definition in C which cannot be implemented with the stable ABI?
Pretty much all the type definitions in CPython except the ones in https://github.com/python/cpython/blob/master/Modules/xxlimited.c will fail on the stable ABI :) It's not that they *can't* be ported to the stable ABI, it's that they *haven't* been, and there isn't currently any kind of code generator to automate the conversion process. For the standard library, the lack of motivation comes from the fact that we recompile for every version anyway, so there's nothing specific to be gained from switching to compiling optional extension modules under the stable ABI instead of the default CPython API. For third party projects, the problem is that they need to continue using static type declarations if they want to support Python 2.7, so using static type declarations for both Py2 and Py3 is a more attractive option than defining their types differently depending on the version. As folks start dropping Python 2.7 support *then* the stable ABI starts to become a more attractive option, as it should let them significantly reduce the number of wheels they publish to PyPI *without* having to maintain two different ways of defining types (assuming we redefine the stable ABI compatibility tags to let people specify a minimum required version that's higher than 3.2). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 13 Jul 2017, at 13:46, Victor Stinner <victor.stinner@gmail.com> wrote:
2017-07-12 20:51 GMT+02:00 Brett Cannon <brett@python.org>:
I also think the motivation doesn't have to be performance but simply cleaning up how we expose our C APIs to users as shown by the fact we have messed up the stable API by making it opt-out instead of opt-in.
It's hard to sell a "cleanup" to users with no carrot :-) Did someone remind trying to sell the "Python 3 cleanup"? :-)
But then there should actually be a carrot ;-). Just declaring the contents of object definitions private in the documentation could also help, especially when adding preprocessor guards to enable access to those definitions. Consulting adults etc…
Yeah, that's also what I initially thought. Use a cascading hierarchy so that people know they should put anything as high up as possible to minimize its exposure.
Yeah, maybe we can do that.
I have to make my own experiment to make sure that #include doesn't leak symbols by mistakes and that it's still possible to use optimized macros or functions in builtin modules.
Avoiding symbol leaks with a cascading hierarchy should be easy enough, some care may be needed to be able to override definitions in the “more private” headers, especially when making current macros available as functions in the more public headers. Although it could be considered to just remove macros like PyTuple_GET_ITEM from the most public layer.
I think Victor has long-term plans to try and hide the struct details at a higher-level and so that would make macros a bad thing. But ignoring the specific Py_INCREF/DECREF example, switching to functions does buy us the ability to actually change the function implementations between Python versions compared to having to worry about what a macro used to do (which is a possibility with the stable ABI).
I think that my PEP is currently badly written :-)
In fact, the idea is just to make the stable ABI usable :-) Instead of hiding structures *and* remove macros, my idea is just to hide structures but still provides macros... as functions. Basically, it will be the same API, but usable on more various implementations of Python.
It might be better to push users towards tools like ctypes and cffi, the latter especially is tuned to work both with CPython and PyPy and appears to gain momentum. That won’t work for everything, but could work for a large subset of extensions. Ronald
On 12 Jul 2017, at 20:51, Brett Cannon <brett@python.org> wrote:
On Wed, 12 Jul 2017 at 01:25 Ronald Oussoren <ronaldoussoren@mac.com <mailto:ronaldoussoren@mac.com>> wrote:
On 11 Jul 2017, at 12:19, Victor Stinner <victor.stinner@gmail.com <mailto:victor.stinner@gmail.com>> wrote:
Hi,
This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed.
I’m not sure if hiding implementation details will help a lot w.r.t. making CPython more efficient, but cleaning up the public API would avoid accidentally depending on non-public information (and is sound engineering anyway). That said, a lot of care should be taken to avoid breaking existing extensions as the ease of writing extensions is one of the strong points of CPython.
I also think the motivation doesn't have to be performance but simply cleaning up how we expose our C APIs to users as shown by the fact we have messed up the stable API by making it opt-out instead of opt-in.
I agree with this. […]
Step 3: first pass of implementation detail removal ---------------------------------------------------
Modify the ``python`` API:
* Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory.
In this particular case (Py_INCREF/DECREF) making them functions isn’t really useful and is likely to be harmful for performance. It is not useful because these macros manipulate state in a struct that must be public because that struct is included into the structs for custom objects (PyObject_HEAD). Having them as macro’s also doesn’t preclude moving to indirect reference counts. Moving to anything that isn’t reference counts likely needs changes to the API (but not necessarily, see PyPy’s cpext).
I think Victor has long-term plans to try and hide the struct details at a higher-level and so that would make macros a bad thing. But ignoring the specific Py_INCREF/DECREF example, switching to functions does buy us the ability to actually change the function implementations between Python versions compared to having to worry about what a macro used to do (which is a possibility with the stable ABI).
I don’t understand. Moving too functions instead of macros for some thing doesn’t really help with keeping the public API stable (for the non-stable ABI). Avoiding macros does help with keeping more of the object internals hidden, and possibly easier to change within a major release, but doesn’t help (or hinder) changing the implementation of an API. AFAIK there is no API stability guarantee for the details of the struct definitions for object representation, which is why it was possible to change the dict representation for CPython 3.6, and the str representation earlier. I wouldn’t mind having to explicitly opt-in to getting access to those internals, but removing them from public headers altogether does have a cost.
* Slowly remove more and more implementation details from this API.
Modifications of these API should be driven by tests of popular third party packages like:
* Django with database drivers * numpy * scipy * Pillow * lxml * etc.
Compilation errors on these extensions are expected. This step should help to draw a line for the backward incompatible change.
This could also help to find places where the documented API is not sufficient. One of the places where I poke directly into implementation details is a C-level subclass of str (PyUnicode_Type). I’d prefer not doing that, but AFAIK there is no other way to be string-like to the C API other than by being a subclass of str.
Yeah, this would allow us to very clearly know what should or should not be documented (I would say the same for the stdlib but we all know old code didn't hide things with a leading underscore consistently).
I tried to write about how this could help to evolve the API by exposing documented APIs or features for things where extensions currently directly peek and poke into implementation details. Moving away from private stuff is a lot easier when there are sanctioned alternatives :-)
BTW. The reason I need to subclass str: in PyObjC I use a subclass of str to represent Objective-C strings (NSString/NSMutableString), and I need to keep track of the original value; mostly because there are some Objective-C APIs that use object identity. The worst part is that fully initialising the PyUnicodeObject fields often isn’t necessary as a lot of Objective-C strings aren’t used as strings in Python code.
Enhancements becoming possible thanks to a new C API ====================================================
Indirect Reference Counting ---------------------------
* Replace ``Py_ssize_t ob_refcnt;`` (integer) with ``Py_ssize_t *ob_refcnt;`` (pointer to an integer). * Same change for GC headers? * Store all reference counters in a separated memory block (or maybe multiple memory blocks)
This could be done right now with a minimal change to the API: just make the ob_refcnt and ob_type fields of the PyObject struct private by renaming them, in Py3 the documented way to access theses fields is through function macros and these could by changed to do indirect refcounting instead.
I think this is why Victor wants functions, because even if you change the names the macros will be locked into their implementations if you try to write code that supports multiple versions and so you can't change it per-version of Python.
I really don’t understand. The macros are part of the code for a version of Python and can be changed when necessary between python versions; the only advantage of functions is that its easier to tweak the implementation in patch releases. BTW. As I mentioned before the PyObject struct is one that cannot be made private without major changes because that struct is included in all extension object definitions by way of PyObject_HEAD. But anyway, that’s just a particular example and doesn’t mean we cannot hide any implementation details. Ronald P.S. I’ve surfaced because I’m at EuroPython, and experience learns that I’ll likely submerge again afterwards even if I’d prefer not to do so :-(
On Thu, 13 Jul 2017 at 09:12 Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 12 Jul 2017, at 20:51, Brett Cannon <brett@python.org> wrote:
On Wed, 12 Jul 2017 at 01:25 Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 11 Jul 2017, at 12:19, Victor Stinner <victor.stinner@gmail.com> wrote:
[SNIP]
Step 3: first pass of implementation detail removal ---------------------------------------------------
Modify the ``python`` API:
* Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory.
In this particular case (Py_INCREF/DECREF) making them functions isn’t really useful and is likely to be harmful for performance. It is not useful because these macros manipulate state in a struct that must be public because that struct is included into the structs for custom objects (PyObject_HEAD). Having them as macro’s also doesn’t preclude moving to indirect reference counts. Moving to anything that isn’t reference counts likely needs changes to the API (but not necessarily, see PyPy’s cpext).
I think Victor has long-term plans to try and hide the struct details at a higher-level and so that would make macros a bad thing. But ignoring the specific Py_INCREF/DECREF example, switching to functions does buy us the ability to actually change the function implementations between Python versions compared to having to worry about what a macro used to do (which is a possibility with the stable ABI).
I don’t understand. Moving too functions instead of macros for some thing doesn’t really help with keeping the public API stable (for the non-stable ABI).
Sorry, I didn't specify which ABI/API I was talking about; my point was from the stable ABI. I think this is quickly showing how naming is going to play into this since e.g. we say "stable ABI" but call it "Py_LIMITED_API" in the code which is rather confusing. Just to make sure I'm not missing anything, it seems we have a few levels here: 1. The stable A**B**I which is compatible across versions 2. A stable A**P**I which hides enough details that if we change a struct your code won't require an update, just a recompile 3. An API that exposes CPython-specific details such as structs and other details that might not be entirely portable to e.g. PyPy easily but that we try not to break 4. An internal API that we use for implementing the interpreter but don't expect anyone else to use, so we can break it between feature releases (although if e.g. Cython chooses to use it they can) (There's also an API local to a single file, but since that is never exported to the linker it doesn't come into play here.) So, a portable API/ABI, a stable API, a CPython API, and then an internal/core/interpreter API. Correct?
On 14 July 2017 at 02:29, Brett Cannon <brett@python.org> wrote:
On Thu, 13 Jul 2017 at 09:12 Ronald Oussoren <ronaldoussoren@mac.com> wrote:
I don’t understand. Moving too functions instead of macros for some thing doesn’t really help with keeping the public API stable (for the non-stable ABI).
Sorry, I didn't specify which ABI/API I was talking about; my point was from the stable ABI.
I think this is quickly showing how naming is going to play into this since e.g. we say "stable ABI" but call it "Py_LIMITED_API" in the code which is rather confusing.
I honestly think we should just change that symbol to Py_STABLE_ABI (with Py_LIMITED_API retained as a backwards compatibility feature). Yes, Py_LIMITED_API is technically more correct, but it's confusing in practice, while "Py_STABLE_ABI" matches the user's intent: "make sure my binaries only depend on the stable ABI".
Just to make sure I'm not missing anything, it seems we have a few levels here:
1. The stable A**B**I which is compatible across versions 2. A stable A**P**I which hides enough details that if we change a struct your code won't require an update, just a recompile
I don't think we want to promise that the portable API will be completely backwards compatible over time - unlike the stable ABI, it should be subject to Python's normal deprecation policy (i.e. if an API emits a deprecation warning in X.Y, we may remove it entirely in X.Y+1). Instead, I think the key promises of the portable API should be: 1. It only exposes interfaces that are genuinely portable across at least CPython and PyPy 2. It adheres as closely to the stable ABI as it can, with additions made *solely* to support the building of existing popular extension modules (e.g. by adding back static type declaration support)
3. An API that exposes CPython-specific details such as structs and other details that might not be entirely portable to e.g. PyPy easily but that we try not to break 4. An internal API that we use for implementing the interpreter but don't expect anyone else to use, so we can break it between feature releases (although if e.g. Cython chooses to use it they can)
(There's also an API local to a single file, but since that is never exported to the linker it doesn't come into play here.)
So, a portable API/ABI, a stable API, a CPython API, and then an internal/core/interpreter API. Correct?
Not quite: - stable ABI (strict extension module compatibility policy) - portable API (no ABI stability guarantees, normal deprecation policy) - public CPython API (no cross-implementation portability guarantees) - internal-only CPython core API (arbitrary changes, no deprecation warnings) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Victor Stinner schrieb am 11.07.2017 um 12:19:
Split the ``Include/`` directory of CPython:
* ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI [...] Step 3: first pass of implementation detail removal ---------------------------------------------------
Modify the ``python`` API:
* Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory. * Slowly remove more and more implementation details from this API.
From a Cython perspective, it's (not great but) ok if these "implementation details" were moved somewhere else, but it would be a problem if they became entirely unavailable for external modules. Cython uses some of the internals for performance reasons, and we adapt it to changes of these internals whenever necessary.
The question then arises if this proposal fulfills its intended purpose if Cython based tools like NumPy or lxml continued to use internal implementation details in their Cython generated C code. Specifically because that code is generated, I find it acceptable that it actively exploits non-portable details, because it already takes care of adapting to different Python platforms anyway. Cython has incorporated support for CPython, PyPy and Pyston that way, adding others is probably not difficult, and optimising for a specific one (usually CPython) is also easy. The general rule of thumb in Cython core development is that it's ok to exploit internals as long as there is a generic fallback through some C-API operations which can be used in other Python implementations. I'd be happy if that continued to be supported by CPython in the future. Exposing CPython internals is a good thing! :) Stefan
On 13 July 2017 at 08:23, Stefan Behnel <stefan_ml@behnel.de> wrote:
The general rule of thumb in Cython core development is that it's ok to exploit internals as long as there is a generic fallback through some C-API operations which can be used in other Python implementations. I'd be happy if that continued to be supported by CPython in the future. Exposing CPython internals is a good thing! :)
+1 This is my major motivation for suggesting "Include/cpython/" as the directory for the header files that define a supported API that is specific to CPython - it helps make it clear to other implementations that it's OK to go beyond the portable Python C API, but such API extensions should be clearly flagged as implementation specific so that consumers can make an informed decision as to which level they want to target. I do want to revise my naming suggestions slightly though: I think it would make sense for the internal APIs (the ones already guarded by Py_BUILD_CORE) to be under "Include/_core/", where the leading underscore helps to emphasise "if you are not working on CPython itself, you should not be going anywhere near these header files". I think the other key point to clarify will be API versioning, since that will flow through to things like the C ABI compatibility tags in the binary wheel format. Currently [1], that looks like: cp35m # Specifically built for CPython 3.5 with PyMalloc cp35dm # Debugging enabled cp3_10m # Disambiguation uses underscores pp18 # It's the implementation version, not the Python version There's currently only one tag for the stable ABI: abi3 # Built for the stable ABI as of Python 3.2 So I think the existing Py_LIMITED_API/stable ABI is the right place for the strict "No public structs!" policy that completely decouples extension modules from CPython internals. We'll just need to refine the definition of the compatibility tags so that folks can properly indicate the minimum required version of that API: abi3 # Py_LIMITED_API=0x03020000 abi32 # Py_LIMITED_API=0x03020000 abi33 # Py_LIMITED_API=0x03030000 abi34 # Py_LIMITED_API=0x03040000 abi35 # Py_LIMITED_API=0x03050000 etc... Where Py_PORTABLE_API would come in is that it could be less strict on the "no public structs" rule (allowing some structs to be exposed as needed to enable building key projects like NumPy and lxml), and instead represent an API that offered source code and extension module portability across Python implementations, rather than strict ABI stability across versions. api37 # Py_PORTABLE_API=0x03070000 api38 # Py_PORTABLE_API=0x03080000 api39 # Py_PORTABLE_API=0x03090000 api3_10 # Py_PORTABLE_API=0x030A0000 etc... Cheers, Nick. [1] https://www.python.org/dev/peps/pep-0425/#details -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2017-07-13 0:23 GMT+02:00 Stefan Behnel <stefan_ml@behnel.de>:
From a Cython perspective, it's (not great but) ok if these "implementation details" were moved somewhere else, but it would be a problem if they became entirely unavailable for external modules. Cython uses some of the internals for performance reasons, and we adapt it to changes of these internals whenever necessary.
I don't want to break the Python world, or my project will just fail. For me, it's ok if Cython or even numpy use the full CPython C API with structures and macros to get the best performances. But we need something like the PEP 399 for C extensions: https://www.python.org/dev/peps/pep-0399/ The best would be if C extensions would have two compilations mode: * "Optimize for CPython" (with impl. detail) * "Use the smaller portable C API" (no impl. detail) For example, use the new private Python 3.6 _PyObject_FastCall() if available, but fallback on PyObject_Call() (or other similar function) otherwise. Once we are are to compile in the two "modes", it becomes possible to run benchmarks and decide if it's worth it. For extensions written with Cython, I expect that Cython will take care of that. The problem is more for C code written manually. The best would be to limit as much as possible to "optimized code" and mostly write "portable" code. Victor
participants (9)
-
Barry Scott -
Brett Cannon -
Eric Snow -
Nick Coghlan -
Paul Moore -
Ronald Oussoren -
Stefan Behnel -
Terry Reedy -
Victor Stinner