On 24 May 2022, at 14:28, Petr Viktorin encukou@gmail.com wrote:
Hello, When reviewing the PyType_FromMetaclass function proposed in [PR 93012], it occurred to me that it would be good to add better support for extending types with opaque structs. Consider the [tutorial example] of extending
list, adapted to PyType_Spec:typedef struct { PyListObject list; int state; } SubListObject;
static PyType_Spec Sublist_spec = { .name = "sublist.SubList", .basicsize = sizeof(SubListObject), ... };
If the PyListObject struct is not available (or opaque), as in the stable ABI, this approach won't work. The practical issue in PR 93012 is with metaclasses, which extend PyTypeObject or PyHeapTypeObject. Same idea but a bit “too meta” to be a good example. “Binding generator”-type projects that use the limited API resort to hacks in this area: see [PySide] or the experimental [limited-API branch of nanobind].
I propose adding API that treats the subclass-specific data as a separate struct, and the “base” as an opaque blob (manipulated either by accessor functions or, if the struct is available, directly). Concretely:
- PyType_Spec.basicsize can be a negative (or zero) N to request -N bytes of *extra* storage on top of what the base class needs, so that the PyType_Spec can be static & read-only.
- a new
void* PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls)returns the pointer to data specific tocls.- a new
Py_ssize_t PyObject_GetTypeataSize(PyTypeObject *cls)returns the size of that data (computed as cls->tp_basicsize - cls->tp_base->tp_basicsize).- itemsize would get similar treatment, but I'll leave details to the PEP
Intended usage:
typedef struct { int state; } SubListObject;
static PyType_Spec Sublist_spec = { .name = "sublist.SubList", .basicsize = -sizeof(SubListObject), ... };
...
sublist_type = PyType_FromSpec(Sublist_spec);
...
SubListObject *data = PyObject_GetTypeData(instance, sublist_type); data->state++;
What do you think? Is it a PEP-worthy idea?
I like this idea. This removes the need to expose class internals for some scenario’s, and would longer term open up opportunities for removing PyObject_HEAD as public API. It should also be easy to implement this efficiently, at least when keeping current restrictions when using multiple inheritance.
Ronald —
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/