Mailman 3 May 2022 - capi-sig

C-API for extending opaque types
by Petr Viktorin 26 May '22

26 May '22

Hello, When reviewing the PyType_FromMetaclass function proposed in [PR 93012], it occurred to me that it would be good to add better support for extending types with opaque structs. Consider the [tutorial example] of extending `list`, adapted to PyType_Spec: typedef struct { PyListObject list; int state; } SubListObject; static PyType_Spec Sublist_spec = { .name = "sublist.SubList", .basicsize = sizeof(SubListObject), ... }; If the PyListObject struct is not available (or opaque), as in the stable ABI, this approach won't work. The practical issue in PR 93012 is with metaclasses, which extend PyTypeObject or PyHeapTypeObject. Same idea but a bit “too meta” to be a good example. “Binding generator”-type projects that use the limited API resort to hacks in this area: see [PySide] or the experimental [limited-API branch of nanobind]. I propose adding API that treats the subclass-specific data as a separate struct, and the “base” as an opaque blob (manipulated either by accessor functions or, if the struct is available, directly). Concretely: - PyType_Spec.basicsize can be a negative (or zero) N to request -N bytes of *extra* storage on top of what the base class needs, so that the PyType_Spec can be static & read-only. - a new `void* PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls)` returns the pointer to data specific to `cls`. - a new `Py_ssize_t PyObject_GetTypeataSize(PyTypeObject *cls)` returns the size of that data (computed as cls->tp_basicsize - cls->tp_base->tp_basicsize). - itemsize would get similar treatment, but I'll leave details to the PEP Intended usage: typedef struct { int state; } SubListObject; static PyType_Spec Sublist_spec = { .name = "sublist.SubList", .basicsize = -sizeof(SubListObject), ... }; ... sublist_type = PyType_FromSpec(Sublist_spec); ... SubListObject *data = PyObject_GetTypeData(instance, sublist_type); data->state++; What do you think? Is it a PEP-worthy idea? [PR 93012]: https://github.com/python/cpython/pull/93012 [tutorial example]: https://docs.python.org/3/extending/newtypes_tutorial.html#subclassing-othe… [PySide]: https://github.com/pyside/pyside2-setup/blob/5.11/sources/shiboken2/libshib… [limited-API branch of nanobind]: https://github.com/wjakob/nanobind/compare/master...limited_api#diff-1457ce…

4 5

Require moderation or move to discuss.python.org?
by Brett Cannon 10 May '22

10 May '22

The signal:noise ratio due to spam is getting a bit high on this mailing list. Would the admins be up for turning on default moderation for people and then selectively turning it off? Or should this group move to discuss.python.org where spam issues aren't as much of a concern?

7 11

New script: add Python 3.10 support to your C extensions without losing Python 3.6 support
by Victor Stinner 07 May '22

07 May '22

Hi, I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 3.6 support: https://github.com/pythoncapi/pythoncapi_compat For example, it replaces "op->ob_type" with "Py_TYPE(op)" and replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)". It relies on the pythoncapi_compat.h header file that I wrote to implement Python 3.9 and Python 3.10 on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference). The PyFrame_GetBack() function was added to Python 3.9 and pythoncapi_compat.h provides it on older Python versions. This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions. This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-) * https://cython.org/ * https://hpy.readthedocs.io/ I hope that my script will facilitate migration of C extensions to HPy. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

2 2

How can we make Python's C-API more friendly for non-C languages?
by Petr Viktorin 06 May '22

06 May '22

In my [C API strategy draft], there wasn't much opposition to the goal of being “easily usable from non-C languages”. But it's clear that as CPython devs we don't have that much relevant experience with those languages. I would like to ask developers of Python bindings, FFIs and alternate C-API implementations for thoughts, suggestions, and pain points to help guide the future of the C-API. I'll collect thoughts in this Google document, though I'm happy to discuss anywhere else: https://docs.google.com/document/d/1fDfF7JanOEyQ9awhKE87-Ajp3H4k7qXL3cmyzPM… I've already asked: - PyPy/cpyext: https://mail.python.org/archives/list/pypy-dev@python.org/thread/44IO4GHWBE… - PyO3 (Rust): https://github.com/PyO3/pyo3/discussions/2346 - nanobind (C++): https://github.com/wjakob/nanobind/discussions/31 If you know more, please let them (or me) know! [C API strategy draft]: https://docs.google.com/document/d/1lrvx-ujHOCuiuqH71L1-nBQFHreI8jsXC966AFu…

5 4