Sorry, folks, but I've been busy the last few days--the Language Summit
is Wednesday, and I had to pack and get myself to SLC for PyCon, &c.
I'll circle back and read the messages on the existing threads
tomorrow. But for now I wanted to post "the wonderful third option" for
forward class definitions we've been batting around for a couple of days.
The fundamental tension in the proposal: we want to /allocate/ the
object at "forward class" time so that everyone can take a reference to
it, but we don't want to /initialize/ the class (e.g. run the class
body) until "continue class" time. However, the class might have a
metaclass with a custom __new__, which would be responsible for
allocating the object, and that isn't run until after the "class body".
How do we allocate the class object early while still supporting custom
metaclass.__new__ calls?
So here's the wonderful third idea. I'm going to change the syntax and
semantics a little, again because we were batting them around quite a
bit, so I'm going to just show you our current thinking.
The general shape of it is the same. First, we have some sort of
forward declaration of the class. I'm going to spell it like this:
forward class C
just for clarity in the discussion. Note that this spelling is also viable:
class C
That is, a "class" statement without parentheses or a colon. (This is
analogous to how C++ does forward declarations of classes, and it was
survivable for them.) Another viable spelling:
C = ForwardClass()
This spelling is nice because it doesn't add new syntax. But maybe it's
less obvious what is going on from a user's perspective.
Whichever spelling we use here, the key idea is that C is bound to a
"ForwardClass" object. A "ForwardClass" object is /not/ a class, it's a
forward declaration of a class. (I suspect ForwardClass is similar to a
typing.ForwardRef, though I've never worked with those so I couldn't say
for sure.) Anyway, all it really has is a name, and the promise that it
might get turned into a class someday. To be explicit about it,
"isinstance(C, type)" is False.
I'm also going to call instances of ForwardClass "immutable". C won't
be immutable forever, but for now you're not permitted to set or change
attributes of C.
Next we have the "continue" class statement. I'm going to spell it like
this:
continue class C(BaseClass, ..., metaclass=MyMetaclass):
# class body goes here
...
I'll mention other possible spellings later. The first change I'll
point out here: we've moved the base classes and the metaclass from the
"forward" statement to the "continue" statement. Technically we could
put them either place if we really cared to. But moving them here seems
better, for reasons you'll see in a minute.
Other than that, this "continue class" statement is similar to what I
(we) proposed before. For example, here C is an expression, not a name.
Now comes the one thing that we might call a "trick". The trick: when
we allocate the ForwardClass instance C, we make it as big as a class
object can ever get. (Mark Shannon assures me this is simply "heap
type", and he knows far more about CPython internals than I ever will.)
Then, when we get to the "continue class" statement, we convince
metaclass.__new__ call to reuse this memory, and preserve the reference
count, but to change the type of the object to "type" (or
what-have-you). C has now been changed from a "ForwardClass" object
into a real type. (Which almost certainly means C is now mutable.)
These semantics let us preserve the entire existing class creation
mechanism. We can call all the same externally-visible steps in the
same externally-visible order. We don't add any new dunder methods, we
don't remove any dunder methods, we don't expose a new dunder attribute
for users to experiment with.
What mechanism do we use to achieve this? metaclass.__new__ always has
to do one of these two things to create the class object: either it
calls "super().__new__", or what we usually call "three-argument type".
In both cases, it passes through the **kwargs that it received into the
super().__new__ call or the three-argument type call. So the "continue
class C" statement will internally add a new kwarg: "__forward__ = C".
If super().__new__ or three-argument type get this kwarg, they won't
allocate a new object, they'll reuse C. They'll preserve the current
reference count, but otherwise overwrite C with all the juicy vitamins
and healthy minerals packed into a Python class object.
So, technically, this means we could spell the "continue class" step
like so:
class C(BaseClass, ..., metaclass=MyMetaClass, __forward__=C):
...
Which means that, combined with the "C = ForwardClass()" statement
above, we could theoretically implement this idea without changing the
syntax of the language. And since we already don't have to change the
underlying semantics of Python class creation, the technical debt
incurred by adding this to the language becomes much smaller.
What could go wrong? My biggest question so far: is there such a thing
as a metaclass written in C, besides type itself? Are there metaclasses
with a __new__ that /doesn't/ call super().__new__ or three-argument
type? If there are are metaclasses that allocate their own class
objects out of raw bytes, they'd likely sidestep this entire process. I
suspect this is rare, if indeed it has ever been done. Anyway, that'd
break this mechanism, so exotic metaclasses like these wouldn't work
with "forward-declared classes". But at least they needn't fail
silently. We just need to add a guard after the call to
metaclass.__new__: if we passed in "__forward__=C" into
metaclass.__new__, and metaclass.__new__ didn't return C, we raise an
exception.
Cheers,
//arry/
p.s. When I say "we" above, I generally mean Eric V. Smith, Barry
Warsaw, Mark Shannon, and myself. But please assume that any dumb ideas
in the proposal are mine, and I was too wrong-headed to listen to the
sage advice from these three wise men when I wrote this email.
