Understanding "is not safe" in typeobject.c
Hi, I'm trying to understand the purpose of the check in tp_new_wrapper() of typeobject.c that results in the "is not safe" exception. I have the following class hierarchy... B -> A -> object ...where B and A are implemented in C. Class A has an implementation of tp_new which does a few context-specific checks before calling PyBaseObject_Type.tp_new() directly to actually create the object. This works fine. However I want to allow class B to be used with a Python mixin. A's tp_new() then has to do something similar to super().__new__(). I have tried to implement this by locating the type object after A in B's MRO, getting it's '__new__' attribute and calling it (using PyObject_Call()) with B passed as the only argument. However I then get the "is not safe" exception, specifically... TypeError: object.__new__(B) is not safe, use B.__new__() I take the same approach for __init__() and that works fine. If I comment out the check in tp_new_wrapper() then everything works fine. So, am I doing something unsafe? If so, what? Or, is the check at fault in not allowing the case of a C extension type with its own tp_new? Thanks, Phil
That code is quite old. This comment tries to explain it: ``` /* Check that the use doesn't do something silly and unsafe like object.__new__(dict). To do this, we check that the most derived base that's not a heap type is this type. */ ``` I think you may have to special-case this and arrange for B.__new__() to be called, like it or not. (If you want us to change the code, please file a bpo bug report. I know that's no fun, but it's the way to get the right people involved.) On Mon, Feb 1, 2021 at 3:27 AM Phil Thompson via Python-Dev < python-dev@python.org> wrote:
Hi,
I'm trying to understand the purpose of the check in tp_new_wrapper() of typeobject.c that results in the "is not safe" exception.
I have the following class hierarchy...
B -> A -> object
...where B and A are implemented in C. Class A has an implementation of tp_new which does a few context-specific checks before calling PyBaseObject_Type.tp_new() directly to actually create the object. This works fine.
However I want to allow class B to be used with a Python mixin. A's tp_new() then has to do something similar to super().__new__(). I have tried to implement this by locating the type object after A in B's MRO, getting it's '__new__' attribute and calling it (using PyObject_Call()) with B passed as the only argument. However I then get the "is not safe" exception, specifically...
TypeError: object.__new__(B) is not safe, use B.__new__()
I take the same approach for __init__() and that works fine.
If I comment out the check in tp_new_wrapper() then everything works fine.
So, am I doing something unsafe? If so, what?
Or, is the check at fault in not allowing the case of a C extension type with its own tp_new?
Thanks, Phil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HRGDEMUR... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
On 2/02/21 12:13 am, Phil Thompson via Python-Dev wrote:
TypeError: object.__new__(B) is not safe, use B.__new__()
It's not safe because object.__new__ doesn't know about any C-level initialisation that A or B need. At the C level, there is always a *single* inheritance hierarchy. The right thing is for B's tp_new to directly call A's tp_new, which calls object's tp_new. Don't worry about Python-level multiple inheritance; the interpreter won't let you create an inheritance structure that would mess this up. -- Greg
On 01/02/2021 23:50, Greg Ewing wrote:
On 2/02/21 12:13 am, Phil Thompson via Python-Dev wrote:
TypeError: object.__new__(B) is not safe, use B.__new__()
It's not safe because object.__new__ doesn't know about any C-level initialisation that A or B need.
But A.__new__ is calling object.__new__ and so can take care of its own needs after the latter returns.
At the C level, there is always a *single* inheritance hierarchy.
Why?
The right thing is for B's tp_new to directly call A's tp_new, which calls object's tp_new.
I want my C-implemented class's __new__ to support cooperative multi-inheritance so my A class cannot assume that object.__new__ is the next in the MRO. I did try to call the next-in-MRO's tp_new directly (rather that calling it's __new__ attribute) but that gave me recursion errors.
Don't worry about Python-level multiple inheritance; the interpreter won't let you create an inheritance structure that would mess this up.
Phil
On 01/02/2021 19:06, Guido van Rossum wrote:
That code is quite old. This comment tries to explain it: ``` /* Check that the use doesn't do something silly and unsafe like object.__new__(dict). To do this, we check that the most derived base that's not a heap type is this type. */ ```
I understand what it is checking, but I don't understand why it is "silly and unsafe".
I think you may have to special-case this and arrange for B.__new__() to be called, like it or not.
But it's already been called. The check fails when trying to subsequently call object.__new__().
(If you want us to change the code, please file a bpo bug report. I know that's no fun, but it's the way to get the right people involved.)
Happy to do that but I first wanted to check if I was doing something "silly" - I'm still not sure. Phil
On Mon, Feb 1, 2021 at 3:27 AM Phil Thompson via Python-Dev < python-dev@python.org> wrote:
Hi,
I'm trying to understand the purpose of the check in tp_new_wrapper() of typeobject.c that results in the "is not safe" exception.
I have the following class hierarchy...
B -> A -> object
...where B and A are implemented in C. Class A has an implementation of tp_new which does a few context-specific checks before calling PyBaseObject_Type.tp_new() directly to actually create the object. This works fine.
However I want to allow class B to be used with a Python mixin. A's tp_new() then has to do something similar to super().__new__(). I have tried to implement this by locating the type object after A in B's MRO, getting it's '__new__' attribute and calling it (using PyObject_Call()) with B passed as the only argument. However I then get the "is not safe" exception, specifically...
TypeError: object.__new__(B) is not safe, use B.__new__()
I take the same approach for __init__() and that works fine.
If I comment out the check in tp_new_wrapper() then everything works fine.
So, am I doing something unsafe? If so, what?
Or, is the check at fault in not allowing the case of a C extension type with its own tp_new?
Thanks, Phil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HRGDEMUR... Code of Conduct: http://python.org/psf/codeofconduct/
On 3/02/21 12:07 am, Phil Thompson wrote:
On 01/02/2021 23:50, Greg Ewing wrote:
At the C level, there is always a *single* inheritance hierarchy.
Why?
Because a C struct can only extend one other C struct.
I want my C-implemented class's __new__ to support cooperative multi-inheritance
I don't think this is possible. Here is what the C API docs have to say about the matter: ----------------------------------------------------------------------- Note If you are creating a co-operative tp_new (one that calls a base type’s tp_new or __new__()), you must not try to determine what method to call using method resolution order at runtime. Always statically determine what type you are going to call, and call its tp_new directly, or via type->tp_base->tp_new. If you do not do this, Python subclasses of your type that also inherit from other Python-defined classes may not work correctly. (Specifically, you may not be able to create instances of such subclasses without getting a TypeError.) ----------------------------------------------------------------------- (Source: https://docs.python.org/3.5/extending/newtypes.html) This doesn't mean that your type can't be used in multiple inheritance, just that __new__ methods in particular can't be cooperative. -- Greg
On 02/02/2021 14:18, Greg Ewing wrote:
On 3/02/21 12:07 am, Phil Thompson wrote:
On 01/02/2021 23:50, Greg Ewing wrote:
At the C level, there is always a *single* inheritance hierarchy.
Why?
Because a C struct can only extend one other C struct.
Yes - I misunderstood what you meant by "at the C level".
I want my C-implemented class's __new__ to support cooperative multi-inheritance
I don't think this is possible. Here is what the C API docs have to say about the matter:
-----------------------------------------------------------------------
Note
If you are creating a co-operative tp_new (one that calls a base type’s tp_new or __new__()), you must not try to determine what method to call using method resolution order at runtime. Always statically determine what type you are going to call, and call its tp_new directly, or via type->tp_base->tp_new. If you do not do this, Python subclasses of your type that also inherit from other Python-defined classes may not work correctly. (Specifically, you may not be able to create instances of such subclasses without getting a TypeError.)
-----------------------------------------------------------------------
(Source: https://docs.python.org/3.5/extending/newtypes.html)
This doesn't mean that your type can't be used in multiple inheritance, just that __new__ methods in particular can't be cooperative.
Thanks - that's fairly definitive, although I don't really understand why __new__ has this particular requirement. Phil
Hi Phil, Hi List, unfortunately you do not give enough code to reproduce what you are doing, but just guessing roughly: you say that you have a hierarchy like B -> A -> object, with B and A implemented in C, and then want to use B with a mixin. Programmers with a non-python background then often write class MyClass(B, Mixin): "whatever" this leads to an MRO of MyClass -> B -> Mixin -> A -> object. This is horror if B and A are written in C, because suddenly B needs to do something with Python code if it wants to have to do something with its superclass Mixin, like creating a new object. I am just guessing that this is what your code tries to do. And this is what the comment considers silly. With class MyClass(Mixin, B): "whatever" there is no problem at all. We get an MRO of MyClass -> Mixin -> B -> A -> object. There is no need for B to do anything special, being written in C it already knows by itself how to construct A as well, no need to fiddle with Python at all. In general, it is usually not necessary to deal with super() in C code at all. The statement that there is only single inheritance on the C level becomes obvious once you look at the MRO: that one is always linear, it is always effectively a single inheritance. This is also why you have to call super() only once even if you have multiple superclasses: super() just follows the effective single inheritance of the MRO. Hope that helps. Cheers Martin
On 3/02/21 4:52 am, Phil Thompson wrote:
Thanks - that's fairly definitive, although I don't really understand why __new__ has this particular requirement.
The job of tp_new is to initialise the C struct. To do this, it first has to initialise the fields of the struct it inherits from, then initialise any fields of its own that it adds, in that order. Initialising the inherited fields must be done by calling the tp_new for the struct that it inherits from. You don't want to call the tp_new of some other class that might have got inserted into the MRO, because you have no idea what kind of C struct it expects to get. Cooperative calling is a nice idea, but it requires rather special conditions to make it work. All the methods must have exactly the same signature, and it mustn't matter what order they're called in. Those conditions don't apply to __new__, especially at the C level where everything is much more strict type-wise. -- Greg
On 3/02/21 11:05 am, Martin Teichmann wrote:
class MyClass(B, Mixin): "whatever"
this leads to an MRO of MyClass -> B -> Mixin -> A -> object.
If you do the tp_new stuff correctly at the C level, you can still create such a class. The only limitation is that if Mixin has a __new__ method written in Python, it won't get called. So if Mixin needs to do any initialisation, it will have to be in __init__, *and* all the __init__ methods in the chain will need to be designed for cooperative calling. -- Greg
Hi Greg, Hi List, Greg wrote:
class MyClass(B, Mixin):
"whatever"
this leads to an MRO of MyClass -> B -> Mixin -> A -> object.
If you do the tp_new stuff correctly at the C level, you can still create such a class. The only limitation is that if Mixin has a __new__ method written in Python, it won't get called.
Yes, it is indeed possible to write a class like this. But that would be, as mentioned in the comment, silly. Now there is nothing bad with being silly once in a while, especially while programming that cannot be avoided at times, I am just warning to use such a construct in a wider context. Because the limitation is actually much more general: if the Mixin has any method written in Python, it won't get called, unless you do a lot of heavy lifting. Now why do you write something in C at all? Two reasons: once for speed, which we can effectively exclude here because the super() construct is slow, or because you want to interact with a library written in another language. Unless your library knows about cooperative inheritance, which is very unlikely, the code from class B is not aware that it is not supposed to call the method inherited from A, but the one from the Mixin instead. So by writing class MyClass(B, Mixin): "whatever" I have major problems to understand what you actually want to achieve, what you could not achieve by writing class MyClass(Mixin, B): "whatever" Since one cannot achieve anything with the former that could not be achieved by the latter, just with much less hassle, I would call the former code silly. That said, I might be wrong and there is indeed something one can achieve with the former but not with the latter, I would be very pleased to hear about this, so please come forward! (For sure, all is based on B and A being written in C, not Python) Cheers Martin
On 02/02/2021 23:08, Greg Ewing wrote:
On 3/02/21 4:52 am, Phil Thompson wrote:
Thanks - that's fairly definitive, although I don't really understand why __new__ has this particular requirement.
The job of tp_new is to initialise the C struct. To do this, it first has to initialise the fields of the struct it inherits from, then initialise any fields of its own that it adds, in that order.
Understood.
Initialising the inherited fields must be done by calling the tp_new for the struct that it inherits from. You don't want to call the tp_new of some other class that might have got inserted into the MRO, because you have no idea what kind of C struct it expects to get.
I had assumed that some other magic in typeobject.c (eg. conflicting meta-classes) would have raised an exception before getting to this stage if there was a conflict.
Cooperative calling is a nice idea, but it requires rather special conditions to make it work. All the methods must have exactly the same signature, and it mustn't matter what order they're called in. Those conditions don't apply to __new__, especially at the C level where everything is much more strict type-wise.
Thanks for the explanation. Phil
On 3/02/21 10:35 pm, Phil Thompson wrote:
On 02/02/2021 23:08, Greg Ewing wrote:
you have no idea what kind of C struct it expects to get.
I had assumed that some other magic in typeobject.c (eg. conflicting meta-classes) would have raised an exception before getting to this stage if there was a conflict.
"No idea" is probably an exaggeration -- there is checking to ensure that all the C structs involved form a sequence of extensions. But they still need to be initialised in the right order, because __new__ methods for later ones assume that the struct they're extending has already been initialised. -- Greg
participants (4)
-
Greg Ewing -
Guido van Rossum -
Martin Teichmann -
Phil Thompson