Metaclasses and Limited API

Hi,
Since there is a discussion about the limited API and headers. A long time ago I asked about this on python-dev:
It concerns me a bit that the limited API does not seem to support MetaClasses in PyType_FromSpec (or FromSpecWithBase). In particular, last time I checked, it seemed impossible to extend the type struct within such a metaclass [1].
There also seemed to be an issue with PyType_FromSpecWithBases
allocating exactly a base type-object, instead of using tp_alloc
as
PyType_Type.tp_new
would.
This and how to efficiently cache Python objects in methods are the two main concerns I have right now for long-term full adoption of the limited API. (There are other difficulties, but for the ones I am aware, I at least have an idea how to they need to be addressed).
I suppose it may actually be possible to hack around the FromSpec issues, although I am not sure right now. unless subclassing PyType_Type in C is for some reason different and very much discouraged?
Cheers,
Sebastian
[1] And e.g. PySide seemed to "do" this, and got away with it, because there is an additional NULL at the end, as a termination for slots or so. Which for them is unused and they just happen to use.

On 2020-06-08 18:00, Sebastian Berg wrote:
Hi,
Since there is a discussion about the limited API and headers. A long time ago I asked about this on python-dev:
It concerns me a bit that the limited API does not seem to support MetaClasses in PyType_FromSpec (or FromSpecWithBase). In particular, last time I checked, it seemed impossible to extend the type struct within such a metaclass [1].
There also seemed to be an issue with PyType_FromSpecWithBases allocating exactly a base type-object, instead of using
tp_alloc
asPyType_Type.tp_new
would.This and how to efficiently cache Python objects in methods are the two main concerns I have right now for long-term full adoption of the limited API. (There are other difficulties, but for the ones I am aware, I at least have an idea how to they need to be addressed).
I suppose it may actually be possible to hack around the FromSpec issues, although I am not sure right now. unless subclassing PyType_Type in C is for some reason different and very much discouraged?
Cheers,
Sebastian
[1] And e.g. PySide seemed to "do" this, and got away with it, because there is an additional NULL at the end, as a termination for slots or so. Which for them is unused and they just happen to use.
I don't think this is about metaclasses per se: you can create a metaclass in the limited API. Rather, it's about attaching state to class objects—at the C level, i.e. allocating some space rather than putting something in the __dict__. AFAIK, calling tp_alloc from PyType_FromSpecWithBases is problematic, because you can't write a custom tp_alloc for a type without assuming too much about the internals.
This all is indeed a problem, and making it work will need some thinking (esp. with inheritance in mind: a superclass and a subclass will need independent storage). But, since it's not needed in Python's standard library, it's not currently on the top of my TODO list. (The thing I want to focus now is documentation, and then access to module state from slot methods.)
And yes, PySide gets away with the hack. That project also reveals more than its share of bugs in CPython, and sometimes needs to be fixed for new Python releases. It's definitely great that PySide exists and works around the issues! But I wouldn't recommend its techniques for general usage.

On 9 Jun 2020, at 14:50, Petr Viktorin <encukou@gmail.com> wrote:
On 2020-06-08 18:00, Sebastian Berg wrote:
Hi, Since there is a discussion about the limited API and headers. A long time ago I asked about this on python-dev: It concerns me a bit that the limited API does not seem to support MetaClasses in PyType_FromSpec (or FromSpecWithBase). In particular, last time I checked, it seemed impossible to extend the type struct within such a metaclass [1]. There also seemed to be an issue with PyType_FromSpecWithBases allocating exactly a base type-object, instead of using
tp_alloc
asPyType_Type.tp_new
would. This and how to efficiently cache Python objects in methods are the two main concerns I have right now for long-term full adoption of the limited API. (There are other difficulties, but for the ones I am aware, I at least have an idea how to they need to be addressed). I suppose it may actually be possible to hack around the FromSpec issues, although I am not sure right now. unless subclassing PyType_Type in C is for some reason different and very much discouraged? Cheers, Sebastian [1] And e.g. PySide seemed to "do" this, and got away with it, because there is an additional NULL at the end, as a termination for slots or so. Which for them is unused and they just happen to use.I don't think this is about metaclasses per se: you can create a metaclass in the limited API. Rather, it's about attaching state to class objects—at the C level, i.e. allocating some space rather than putting something in the __dict__.
That’s not just a problem for classes, the same is true for subclassing other types for which the instance struct is private (e.g. all builtin types in the limited API).
Is is possible to define a class that uses a metaclass using PyType_FromSpec or one of its variants? AFAIK it is not possible to set the ob_type field of the to be created type using these APIs.
Ronald
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

On 2020-06-09 18:32, Ronald Oussoren wrote:
On 9 Jun 2020, at 14:50, Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 2020-06-08 18:00, Sebastian Berg wrote:
Hi, Since there is a discussion about the limited API and headers. A long time ago I asked about this on python-dev: It concerns me a bit that the limited API does not seem to support MetaClasses in PyType_FromSpec (or FromSpecWithBase). In particular, last time I checked, it seemed impossible to extend the type struct within such a metaclass [1]. There also seemed to be an issue with PyType_FromSpecWithBases allocating exactly a base type-object, instead of using
tp_alloc
asPyType_Type.tp_new
would. This and how to efficiently cache Python objects in methods are the two main concerns I have right now for long-term full adoption of the limited API. (There are other difficulties, but for the ones I am aware, I at least have an idea how to they need to be addressed). I suppose it may actually be possible to hack around the FromSpec issues, although I am not sure right now. unless subclassing PyType_Type in C is for some reason different and very much discouraged? Cheers, Sebastian [1] And e.g. PySide seemed to "do" this, and got away with it, because there is an additional NULL at the end, as a termination for slots or so. Which for them is unused and they just happen to use.I don't think this is about metaclasses per se: you can create a metaclass in the limited API. Rather, it's about attaching state to class objects—at the C level, i.e. allocating some space rather than putting something in the __dict__.
That’s not just a problem for classes, the same is true for subclassing other types for which the instance struct is private (e.g. all builtin types in the limited API).
For classes, the problem is even worse because they're implemented as variable-length objects ("containers", PyVarObject) to hold slots.
For normal objects, you can play tricks with the ob_size of the superclass, which tells you where additional data can be located in the instance. But for classes (and lists, tuples, ...), it's not clear what the memory layout for additional storage should be, let alone how to access/define it.
Is is possible to define a class that uses a metaclass using PyType_FromSpec or one of its variants? AFAIK it is not possible to set the ob_type field of the to be created type using these APIs.
Ah, I see what you mean. No, it's currently not possible to set the metaclass using PyType_FromSpec. I don't see a reason why adding a Py_ob_type slot would be impossible, but I haven't really given it much thought.

On Tue, 2020-06-09 at 20:48 +0200, Petr Viktorin wrote:
On 2020-06-09 18:32, Ronald Oussoren wrote:
On 9 Jun 2020, at 14:50, Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 2020-06-08 18:00, Sebastian Berg wrote:
Hi, Since there is a discussion about the limited API and headers. A long time ago I asked about this on python-dev: It concerns me a bit that the limited API does not seem to support MetaClasses in PyType_FromSpec (or FromSpecWithBase). In particular, last time I checked, it seemed impossible to extend the type struct within such a metaclass [1]. There also seemed to be an issue with PyType_FromSpecWithBases allocating exactly a base type-object, instead of using
tp_alloc
asPyType_Type.tp_new
would. This and how to efficiently cache Python objects in methods are the two main concerns I have right now for long-term full adoption of the limited API. (There are other difficulties, but for the ones I am aware, I at least have an idea how to they need to be addressed). I suppose it may actually be possible to hack around the FromSpec issues, although I am not sure right now. unless subclassing PyType_Type in C is for some reason different and very much discouraged? Cheers, Sebastian [1] And e.g. PySide seemed to "do" this, and got away with it, because there is an additional NULL at the end, as a termination for slots or so. Which for them is unused and they just happen to use.I don't think this is about metaclasses per se: you can create a metaclass in the limited API. Rather, it's about attaching state to class objects—at the C level, i.e. allocating some space rather than putting something in the __dict__.
That’s not just a problem for classes, the same is true for subclassing other types for which the instance struct is private (e.g. all builtin types in the limited API).
For classes, the problem is even worse because they're implemented as variable-length objects ("containers", PyVarObject) to hold slots.
For normal objects, you can play tricks with the ob_size of the superclass, which tells you where additional data can be located in the instance. But for classes (and lists, tuples, ...), it's not clear what the memory layout for additional storage should be, let alone how to access/define it.
Is is possible to define a class that uses a metaclass using PyType_FromSpec or one of its variants? AFAIK it is not possible to set the ob_type field of the to be created type using these APIs.
Ah, I see what you mean. No, it's currently not possible to set the metaclass using PyType_FromSpec. I don't see a reason why adding a Py_ob_type slot would be impossible, but I haven't really given it much thought.
Ah, right, I think that was actually one of main issues... For me this was a bit more tedious. I am currently not too scared of using the limited API for the metaclass (in the end, it probably is a true static, although I am not sure that helps with embedding/sub- interpreters). But it would be much better to use (only) the limited API for dynamically creating the metaclasses instances.
I suppose the other issues are indeed identical to the difficulties of subclassing in general (with extended C-struct/data).
The only thing that would worry me, is if anyone thinks that these things should not or cannot be made to work. I don't mind if we don't have concrete plans yet, nothing about this seems quite concrete anyway.
However, I do want to be the person who maneuvers important C-extension modules into a position where sub-interpreters cannot reasonably made to work at *some* point.
So, I assume that it is OK to expect solutions in the future? I.e. by the time we expect large C-extension modules to put effort into working flawlessly in sub-interpreters?
Cheers,
Sebastian
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: sebastian@sipsolutions.net

On 9 Jun 2020, at 21:12, Sebastian Berg <sebastian@sipsolutions.net> wrote:
On Tue, 2020-06-09 at 20:48 +0200, Petr Viktorin wrote:
On 2020-06-09 18:32, Ronald Oussoren wrote:
On 9 Jun 2020, at 14:50, Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
[…]
I don't think this is about metaclasses per se: you can create a metaclass in the limited API. Rather, it's about attaching state to class objects—at the C level, i.e. allocating some space rather than putting something in the __dict__.
That’s not just a problem for classes, the same is true for subclassing other types for which the instance struct is private (e.g. all builtin types in the limited API).
For classes, the problem is even worse because they're implemented as variable-length objects ("containers", PyVarObject) to hold slots.
For normal objects, you can play tricks with the ob_size of the superclass, which tells you where additional data can be located in the instance. But for classes (and lists, tuples, ...), it's not clear what the memory layout for additional storage should be, let alone how to access/define it.
More indirection could help. One way to do this is to have an API to retrieve a pointer to the data for a specific class. That is, subclasses define a struct with the additional data for that subclass, an API returns a pointer to that data for a given object.
Is is possible to define a class that uses a metaclass using PyType_FromSpec or one of its variants? AFAIK it is not possible to set the ob_type field of the to be created type using these APIs.
Ah, I see what you mean. No, it's currently not possible to set the metaclass using PyType_FromSpec. I don't see a reason why adding a Py_ob_type slot would be impossible, but I haven't really given it much thought.
Ah, right, I think that was actually one of main issues... For me this was a bit more tedious. I am currently not too scared of using the limited API for the metaclass (in the end, it probably is a true static, although I am not sure that helps with embedding/sub- interpreters). But it would be much better to use (only) the limited API for dynamically creating the metaclasses instances.
This is one technical limitation that stops me from moving to the new API in PyObjC.
Ronald
participants (3)
-
Petr Viktorin
-
Ronald Oussoren
-
Sebastian Berg