Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object
Here's one alternate idea for how to implement the "forward class" syntax.
The entire point of the "forward class" statement is that it creates the real actual class object. But what if it wasn't actually the "real" class object? What if it was only a proxy for the real object?
In this scenario, the syntax of "forward object" remains the same. You define the class's bases and metaclass. But all "forward class" does is create a simple, lightweight class proxy object. This object has a few built-in dunder values, __name__ etc. It also allows you to set attributes, so let's assume (for now) it calls metaclass.__prepare__ and uses the returned "dict-like object" as the class proxy object __dict__.
"continue class" internally performs all the rest of the class-creation machinery. (Everything except __prepare__, as we already called that.) The first step is metaclass.__new__, which returns the real class object. "continue class" takes that object and calls a method on the class proxy object that says "here's your real class object". From that moment on, the proxy becomes a pass-through for the "real" class object, and nobody ever sees a reference to the "real" class object ever again. Every interaction with the class proxy object is passed through to the underlying class object. __getattribute__ calls on the proxy look up the attribute in the underlying class object. If the object returned is a bound method object, it rebinds that callable with the class proxy instead, so that the "self" passed in to methods is the proxy object. Both base_cls.__init_subclass__ and cls.__init__ see the proxy object during class creation. As far as Python user code is concerned, the class proxy *is* the class, in every way, important or not.
The upside: this moves all class object creation code into "continue class" call. We don't have to replace __new__ with two new calls.
The downside: a dinky overhead to every interaction with a "forward class" class object and with instances of a "forward class" class object.
A huge concern: how does this interact with metaclasses implemented in C? If you make a method call on a proxy class object, and that calls a C function from the metaclass, we'd presumably have to pass in the "real class object", not the proxy class object. Which means references to the real class object could leak out somewhere, and now we have a real-class-object vs proxy-class-object identity crisis. Is this a real concern?
A possible concern: what if metaclass.__new__ keeps a reference to the object it created? Now we have two objects with an identity crisis. I don't know if people ever do that. Fingers crossed that they don't. Or maybe we add a new dunder method:
@special_cased_staticmethod metaclass.__bind_proxy__(metaclass, proxy, cls)
This tells the metaclass "bind cls to this proxy object", so metaclasses that care can update their database or whatever. The default implementation uses the appropriate mechanism, whatever it is.
One additional probably-bad idea: in the case where it's just a normal "class" statement, and we're not binding it to a proxy, should we call this?
metaclass.__bind_proxy__(metaclass, None, cls)
The idea there being "if you register the class objects you create, do the registration in __bind_proxy__, it's always called, and you'll always know the canonical object in there". I'm guessing probably not, in which case we tell metaclasses that track the class objects we create "go ahead and track the object you return from __new__, but be prepared to update your tracking info in case we call __bind_proxy__ on you".
A small but awfully complicated wrinkle here: what do we do if the metaclass implements __del__? Obviously, we have to call __del__ with the "real" class object, so it can be destroyed properly. But __del__ might resurrect that object, which means someone took a reference to it.
One final note. Given that, in this scenario, all real class creation happens in "continue class", we could move the bases and metaclass declaration down to the "continue class" statement. The resulting syntax would look like:
forward class X
...
continue class X(base1, base2, metaclass=AmazingMeta, rocket="booster")
Is that better? worse? doesn't matter? I don't have an intuition about it right now--I can see advantages to both sides, and no obvious deciding factor. Certainly this syntax prevents us from calling __prepare__ so early, so we'd have to use a real dict in the "forward class" proxy object until we reached continue, then copy the values from that dict into the "dict-like object", etc.
TL;DR (literally, I will go back and read it now, but after reading the first paragraphs: _a proxy_ object yes, then dividing class creation in 2 blocks would not break things)
/me goes back to text.
On Fri, Apr 22, 2022 at 10:20 PM Larry Hastings larry@hastings.org wrote:
Here's one alternate idea for how to implement the "forward class" syntax.
The entire point of the "forward class" statement is that it creates the real actual class object. But what if it wasn't actually the "real" class object? What if it was only a proxy for the real object?
In this scenario, the syntax of "forward object" remains the same. You define the class's bases and metaclass. But all "forward class" does is create a simple, lightweight class proxy object. This object has a few built-in dunder values, __name__ etc. It also allows you to set attributes, so let's assume (for now) it calls metaclass.__prepare__ and uses the returned "dict-like object" as the class proxy object __dict__.
"continue class" internally performs all the rest of the class-creation machinery. (Everything except __prepare__, as we already called that.) The first step is metaclass.__new__, which returns the real class object. "continue class" takes that object and calls a method on the class proxy object that says "here's your real class object". From that moment on, the proxy becomes a pass-through for the "real" class object, and nobody ever sees a reference to the "real" class object ever again. Every interaction with the class proxy object is passed through to the underlying class object. __getattribute__ calls on the proxy look up the attribute in the underlying class object. If the object returned is a bound method object, it rebinds that callable with the class proxy instead, so that the "self" passed in to methods is the proxy object. Both base_cls.__init_subclass__ and cls.__init__ see the proxy object during class creation. As far as Python user code is concerned, the class proxy *is* the class, in every way, important or not.
The upside: this moves all class object creation code into "continue class" call. We don't have to replace __new__ with two new calls.
The downside: a dinky overhead to every interaction with a "forward class" class object and with instances of a "forward class" class object.
A huge concern: how does this interact with metaclasses implemented in C? If you make a method call on a proxy class object, and that calls a C function from the metaclass, we'd presumably have to pass in the "real class object", not the proxy class object. Which means references to the real class object could leak out somewhere, and now we have a real-class-object vs proxy-class-object identity crisis. Is this a real concern?
A possible concern: what if metaclass.__new__ keeps a reference to the object it created? Now we have two objects with an identity crisis. I don't know if people ever do that. Fingers crossed that they don't. Or maybe we add a new dunder method:
@special_cased_staticmethod metaclass.__bind_proxy__(metaclass, proxy, cls)This tells the metaclass "bind cls to this proxy object", so metaclasses that care can update their database or whatever. The default implementation uses the appropriate mechanism, whatever it is.
One additional probably-bad idea: in the case where it's just a normal "class" statement, and we're not binding it to a proxy, should we call this?
metaclass.__bind_proxy__(metaclass, None, cls)The idea there being "if you register the class objects you create, do the registration in __bind_proxy__, it's always called, and you'll always know the canonical object in there". I'm guessing probably not, in which case we tell metaclasses that track the class objects we create "go ahead and track the object you return from __new__, but be prepared to update your tracking info in case we call __bind_proxy__ on you".
A small but awfully complicated wrinkle here: what do we do if the metaclass implements __del__? Obviously, we have to call __del__ with the "real" class object, so it can be destroyed properly. But __del__ might resurrect that object, which means someone took a reference to it.
One final note. Given that, in this scenario, all real class creation happens in "continue class", we could move the bases and metaclass declaration down to the "continue class" statement. The resulting syntax would look like:
forward class X ... continue class X(base1, base2, metaclass=AmazingMeta,rocket="booster")
Is that better? worse? doesn't matter? I don't have an intuition about it right now--I can see advantages to both sides, and no obvious deciding factor. Certainly this syntax prevents us from calling __prepare__ so early, so we'd have to use a real dict in the "forward class" proxy object until we reached continue, then copy the values from that dict into the "dict-like object", etc.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7E... Code of Conduct: http://python.org/psf/codeofconduct/
So -
good idea on creating the proxy.
But would you really __need__ this badly that the proxy object would "become" the new class object, preserving its "id"?
Just name re-binding (and the static type checkers _knowing_ the name will be re-assign to the actuall class object later) seems to be pretty straightforward, and would break too little.
If one assigns the proxy to a different name in the meantime, and keep a reference to it: "consenting adults" apply: one just got a somewhat useless stub object to play along with.
Ok, it would be more of a "stub" object than a "proxy" object,
So, if the "proxy" is really needed - there is one type of existing proxy to classes in Python that is really transparent, and that is not easily creatable by Python code - those are the instances of "super()" - the will answer proper even to isinstance and issubclass calls, as it goes beyond what is possible by customizing __getattribute__. So, if a proxy is really needed instead of a simple name rebind, probably the existing code for "super' can be reused for them.
As for: "do people keep references for the classes inside "__new__"? - Yes, all the time. The "__bind_proxy__" method you describe would again break compatibility (although in a much more reasonable way than the "__new__" split.) But then, the re-binding, if any, could take place inside "type.__new__", before it returns the newly created "cls" to the call to it made inside the custom metaclass.__new__ . Some code might break when getting back a proxied class at this point, but, up to that point, a lot of code could also break with this kind of proxies - "super()" instances work well but there are, of course, lots of corner cases. All in all, I think this is overkill for the problem at hand.
As I wrote before: I stand with what Paul Moore wrote: in an ideal universe annotations for type checking should be optional. In the real world, there is a lot of pressure, surging from everywhere, for strict type-checking in all types of projects, open source or not, and I find this a very sad state of things. Breaking the language compatibility and features because it is needed for "optional" type checking is sad-squared.
On Fri, Apr 22, 2022 at 10:20 PM Larry Hastings larry@hastings.org wrote:
Here's one alternate idea for how to implement the "forward class" syntax.
The entire point of the "forward class" statement is that it creates the real actual class object. But what if it wasn't actually the "real" class object? What if it was only a proxy for the real object?
In this scenario, the syntax of "forward object" remains the same. You define the class's bases and metaclass. But all "forward class" does is create a simple, lightweight class proxy object. This object has a few built-in dunder values, __name__ etc. It also allows you to set attributes, so let's assume (for now) it calls metaclass.__prepare__ and uses the returned "dict-like object" as the class proxy object __dict__.
"continue class" internally performs all the rest of the class-creation machinery. (Everything except __prepare__, as we already called that.) The first step is metaclass.__new__, which returns the real class object. "continue class" takes that object and calls a method on the class proxy object that says "here's your real class object". From that moment on, the proxy becomes a pass-through for the "real" class object, and nobody ever sees a reference to the "real" class object ever again. Every interaction with the class proxy object is passed through to the underlying class object. __getattribute__ calls on the proxy look up the attribute in the underlying class object. If the object returned is a bound method object, it rebinds that callable with the class proxy instead, so that the "self" passed in to methods is the proxy object. Both base_cls.__init_subclass__ and cls.__init__ see the proxy object during class creation. As far as Python user code is concerned, the class proxy *is* the class, in every way, important or not.
The upside: this moves all class object creation code into "continue class" call. We don't have to replace __new__ with two new calls.
The downside: a dinky overhead to every interaction with a "forward class" class object and with instances of a "forward class" class object.
A huge concern: how does this interact with metaclasses implemented in C? If you make a method call on a proxy class object, and that calls a C function from the metaclass, we'd presumably have to pass in the "real class object", not the proxy class object. Which means references to the real class object could leak out somewhere, and now we have a real-class-object vs proxy-class-object identity crisis. Is this a real concern?
A possible concern: what if metaclass.__new__ keeps a reference to the object it created? Now we have two objects with an identity crisis. I don't know if people ever do that. Fingers crossed that they don't. Or maybe we add a new dunder method:
@special_cased_staticmethod metaclass.__bind_proxy__(metaclass, proxy, cls)This tells the metaclass "bind cls to this proxy object", so metaclasses that care can update their database or whatever. The default implementation uses the appropriate mechanism, whatever it is.
One additional probably-bad idea: in the case where it's just a normal "class" statement, and we're not binding it to a proxy, should we call this?
metaclass.__bind_proxy__(metaclass, None, cls)The idea there being "if you register the class objects you create, do the registration in __bind_proxy__, it's always called, and you'll always know the canonical object in there". I'm guessing probably not, in which case we tell metaclasses that track the class objects we create "go ahead and track the object you return from __new__, but be prepared to update your tracking info in case we call __bind_proxy__ on you".
A small but awfully complicated wrinkle here: what do we do if the metaclass implements __del__? Obviously, we have to call __del__ with the "real" class object, so it can be destroyed properly. But __del__ might resurrect that object, which means someone took a reference to it.
One final note. Given that, in this scenario, all real class creation happens in "continue class", we could move the bases and metaclass declaration down to the "continue class" statement. The resulting syntax would look like:
forward class X ... continue class X(base1, base2, metaclass=AmazingMeta,rocket="booster")
Is that better? worse? doesn't matter? I don't have an intuition about it right now--I can see advantages to both sides, and no obvious deciding factor. Certainly this syntax prevents us from calling __prepare__ so early, so we'd have to use a real dict in the "forward class" proxy object until we reached continue, then copy the values from that dict into the "dict-like object", etc.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7E... Code of Conduct: http://python.org/psf/codeofconduct/
On 23. 04. 22 3:15, Larry Hastings wrote:
Here's one alternate idea for how to implement the "forward class" syntax.
The entire point of the "forward class" statement is that it creates the real actual class object. But what if it wasn't actually the "real" class object? What if it was only a proxy for the real object?
In this scenario, the syntax of "forward object" remains the same. You define the class's bases and metaclass. But all "forward class" does is create a simple, lightweight class proxy object. This object has a few built-in dunder values, __name__ etc. It also allows you to set attributes, so let's assume (for now) it calls metaclass.__prepare__ and uses the returned "dict-like object" as the class proxy object __dict__.
"continue class" internally performs all the rest of the class-creation machinery. (Everything except __prepare__, as we already called that.) The first step is metaclass.__new__, which returns the real class object. "continue class" takes that object and calls a method on the class proxy object that says "here's your real class object". From that moment on, the proxy becomes a pass-through for the "real" class object, and nobody ever sees a reference to the "real" class object ever again. Every interaction with the class proxy object is passed through to the underlying class object. __getattribute__ calls on the proxy look up the attribute in the underlying class object. If the object returned is a bound method object, it rebinds that callable with the class proxy instead, so that the "self" passed in to methods is the proxy object. Both base_cls.__init_subclass__ and cls.__init__ see the proxy object during class creation. As far as Python user code is concerned, the class proxy *is* the class, in every way, important or not.
Sadly, I think that if you try to implement this you'll discover a fractal of little issues, each requiring another hack to solve. (But it seems the “main” forward/continue class proposal suffered a similar fate, hasn't it?)
I can see lots of possible issues in interaction with C code (which Python user code would call).
What would the __class__ attribute hold? What would C's ob_type be? What would be in the MROs? How would Exception subclasses work? (taking `except` as an example of something that uses the real inheritance chain rather than __isinstance__/__getattribute__ magic)
The upside: this moves all class object creation code into "continue class" call. We don't have to replace __new__ with two new calls.
The downside: a dinky overhead to every interaction with a "forward class" class object and with instances of a "forward class" class object.
A huge concern: how does this interact with metaclasses implemented in C? If you make a method call on a proxy class object, and that calls a C function from the metaclass, we'd presumably have to pass in the "real class object", not the proxy class object. Which means references to the real class object could leak out somewhere, and now we have a real-class-object vs proxy-class-object identity crisis. Is this a real concern?
A possible concern: what if metaclass.__new__ keeps a reference to the object it created? Now we have two objects with an identity crisis. I don't know if people ever do that. Fingers crossed that they don't. Or maybe we add a new dunder method:
@special_cased_staticmethod metaclass.__bind_proxy__(metaclass, proxy, cls)
This tells the metaclass "bind cls to this proxy object", so metaclasses that care can update their database or whatever. The default implementation uses the appropriate mechanism, whatever it is.
One additional probably-bad idea: in the case where it's just a normal "class" statement, and we're not binding it to a proxy, should we call this?
metaclass.__bind_proxy__(metaclass, None, cls)
The idea there being "if you register the class objects you create, do the registration in __bind_proxy__, it's always called, and you'll always know the canonical object in there". I'm guessing probably not, in which case we tell metaclasses that track the class objects we create "go ahead and track the object you return from __new__, but be prepared to update your tracking info in case we call __bind_proxy__ on you".
A small but awfully complicated wrinkle here: what do we do if the metaclass implements __del__? Obviously, we have to call __del__ with the "real" class object, so it can be destroyed properly. But __del__ might resurrect that object, which means someone took a reference to it.
One final note. Given that, in this scenario, all real class creation happens in "continue class", we could move the bases and metaclass declaration down to the "continue class" statement. The resulting syntax would look like:
forward class X
...
continue class X(base1, base2, metaclass=AmazingMeta, rocket="booster")
Is that better? worse? doesn't matter? I don't have an intuition about it right now--I can see advantages to both sides, and no obvious deciding factor. Certainly this syntax prevents us from calling __prepare__ so early, so we'd have to use a real dict in the "forward class" proxy object until we reached continue, then copy the values from that dict into the "dict-like object", etc.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7E...
Code of Conduct: http://python.org/psf/codeofconduct/
participants (3)
-
Joao S. O. Bueno -
Larry Hastings -
Petr Viktorin