Mailman 3 Proto-PEP part 4: The wonderful third option - Python-Dev

26 Apr 2022

      Sorry, folks, but I've been busy the last few days--the Language Summit 
is Wednesday, and I had to pack and get myself to SLC for PyCon, &c.  
I'll circle back and read the messages on the existing threads 
tomorrow.  But for now I wanted to post "the wonderful third option" for 
forward class definitions we've been batting around for a couple of days.

The fundamental tension in the proposal: we want to /allocate/ the 
object at "forward class" time so that everyone can take a reference to 
it, but we don't want to /initialize/ the class (e.g. run the class 
body) until "continue class" time.  However, the class might have a 
metaclass with a custom __new__, which would be responsible for 
allocating the object, and that isn't run until after the "class body".  
How do we allocate the class object early while still supporting custom 
metaclass.__new__ calls?

So here's the wonderful third idea.  I'm going to change the syntax and 
semantics a little, again because we were batting them around quite a 
bit, so I'm going to just show you our current thinking.

The general shape of it is the same.  First, we have some sort of 
forward declaration of the class.  I'm going to spell it like this:

    forward class C

just for clarity in the discussion.  Note that this spelling is also viable:

    class C

That is, a "class" statement without parentheses or a colon. (This is 
analogous to how C++ does forward declarations of classes, and it was 
survivable for them.)  Another viable spelling:

    C = ForwardClass()

This spelling is nice because it doesn't add new syntax.  But maybe it's 
less obvious what is going on from a user's perspective.

Whichever spelling we use here, the key idea is that C is bound to a 
"ForwardClass" object.  A "ForwardClass" object is /not/ a class, it's a 
forward declaration of a class.  (I suspect ForwardClass is similar to a 
typing.ForwardRef, though I've never worked with those so I couldn't say 
for sure.)  Anyway, all it really has is a name, and the promise that it 
might get turned into a class someday.  To be explicit about it, 
"isinstance(C, type)" is False.

I'm also going to call instances of ForwardClass "immutable".  C won't 
be immutable forever, but for now you're not permitted to set or change 
attributes of C.

Next we have the "continue" class statement.  I'm going to spell it like 
this:

    continue class C(BaseClass, ..., metaclass=MyMetaclass):
         # class body goes here
         ...

I'll mention other possible spellings later.  The first change I'll 
point out here: we've moved the base classes and the metaclass from the 
"forward" statement to the "continue" statement.  Technically we could 
put them either place if we really cared to.  But moving them here seems 
better, for reasons you'll see in a minute.

Other than that, this "continue class" statement is similar to what I 
(we) proposed before.  For example, here C is an expression, not a name.

Now comes the one thing that we might call a "trick".  The trick: when 
we allocate the ForwardClass instance C, we make it as big as a class 
object can ever get.  (Mark Shannon assures me this is simply "heap 
type", and he knows far more about CPython internals than I ever will.)  
Then, when we get to the "continue class" statement, we convince 
metaclass.__new__ call to reuse this memory, and preserve the reference 
count, but to change the type of the object to "type" (or 
what-have-you).  C has now been changed from a "ForwardClass" object 
into a real type.  (Which almost certainly means C is now mutable.)

These semantics let us preserve the entire existing class creation 
mechanism.  We can call all the same externally-visible steps in the 
same externally-visible order.  We don't add any new dunder methods, we 
don't remove any dunder methods, we don't expose a new dunder attribute 
for users to experiment with.

What mechanism do we use to achieve this?  metaclass.__new__ always has 
to do one of these two things to create the class object: either it 
calls "super().__new__", or what we usually call "three-argument type".  
In both cases, it passes through the **kwargs that it received into the 
super().__new__ call or the three-argument type call.  So the "continue 
class C" statement will internally add a new kwarg: "__forward__ = C".  
If super().__new__ or three-argument type get this kwarg, they won't 
allocate a new object, they'll reuse C.  They'll preserve the current 
reference count, but otherwise overwrite C with all the juicy vitamins 
and healthy minerals packed into a Python class object.

So, technically, this means we could spell the "continue class" step 
like so:

    class C(BaseClass, ..., metaclass=MyMetaClass, __forward__=C):
         ...

Which means that, combined with the "C = ForwardClass()" statement 
above, we could theoretically implement this idea without changing the 
syntax of the language.  And since we already don't have to change the 
underlying semantics of Python class creation, the technical debt 
incurred by adding this to the language becomes much smaller.

What could go wrong?  My biggest question so far: is there such a thing 
as a metaclass written in C, besides type itself?  Are there metaclasses 
with a __new__ that /doesn't/ call super().__new__ or three-argument 
type?  If there are are metaclasses that allocate their own class 
objects out of raw bytes, they'd likely sidestep this entire process.  I 
suspect this is rare, if indeed it has ever been done.  Anyway, that'd 
break this mechanism, so exotic metaclasses like these wouldn't work 
with "forward-declared classes".  But at least they needn't fail 
silently.  We just need to add a guard after the call to 
metaclass.__new__: if we passed in "__forward__=C" into 
metaclass.__new__, and metaclass.__new__ didn't return C, we raise an 
exception.

Cheers,

//arry/

p.s. When I say "we" above, I generally mean Eric V. Smith, Barry 
Warsaw, Mark Shannon, and myself.  But please assume that any dumb ideas 
in the proposal are mine, and I was too wrong-headed to listen to the 
sage advice from these three wise men when I wrote this email.

Proto-PEP part 4: The wonderful third option

Larry Hastings

Ronald Oussoren

Larry Hastings

Ronald Oussoren

Guido van Rossum

Carl Meyer

Rob Cliffe

Guido van Rossum

Larry Hastings

Paul Bryan

Larry Hastings

Carl Meyer

Paul Bryan

dw-git＠d-woods.co.uk

Joao S. O. Bueno

Ronald Oussoren

MRAB

Larry Hastings

Chris Angelico

Barry Warsaw

Steven D'Aprano

tags

participants (12)