[Python-Dev] Possible rough edges in Python 3 metaclasses (was Re: Language reference updated for metaclasses)

[Nick Coghlan]

On Wed, Jun 6, 2012 at 1:28 AM, PJ Eby <pje at telecommunity.com> wrote:
> IOW, my motivation for saying, "hey, can't I just use this nice hook here"
> was to avoid asking for a *new* feature, if there weren't enough other
> people interested in a decoration protocol of this sort.
>
> That is, I was trying to NOT make anybody do a bunch of work on my behalf.
> (Clearly, I wasn't successful in the attempt, but I should at least get
> credit for trying.  ;-)

OK, I can accept that. I mainly just wanted to head off the idea of
overloading __build_class__ *anyway*, even if you weren't successful
in getting agreement in bringing back __metaclass__ support, or a
sufficiently powerful replacement mechanism.

The idea of making __build_class__ itself public *has* been discussed,
and the outcome of that discussion was the creation of
types.new_class() as a way to programmatically define classes that
respects the new __prepare__() hook.

> In general, implementing what's effectively inherited decoration is what
> *most* metaclasses actually get used for, so PEP 422 is a big step forward
> in that.

Yeah, that's what I realised (and will try to explain better in the
next version of the PEP, based on my reply to Xavier).

> Sketching something to get a feel for the PEP...
>
> def inheritable(*decos):
>     """Wrap a class with inheritable decorators"""
>     def decorate(cls):
>         cls.__decorators__ =
> list(decos)+list(cls.__dict__.get('__decorators__',()))
>         for deco in reversed(decos):
>             cls = deco(cls)
>         return cls

Yep, that should work.

> Hm.  There are a few interesting consequences of the PEP as written.
> In-body decorators affect the __class__ closure (and thus super()), but
> out-of-body decorators don't.  By me this is a good thing, but it is a bit
> of complexity that needs mentioning.

It's also worth highlighting this as a limitation of the currently
supported metaclass based approach. When a metaclass runs, __class__
hasn't been filled in yet, so any attempts to call methods that use it
(including via zero-argument super()) will fail.

That's why my _register example at [1] ended up relying on the default
argument hack: using __class__ (which is what I tried first without
thinking it through) actually fails, complaining that the cell hasn't
been populated. Under PEP 422, the default argument hack wouldn't be
necessary, you could just write it as:

    def _register(cls):
            __class__._registry.append(cls)
            return cls

[1] https://bitbucket.org/ncoghlan/misc/src/default/pep422.py

>  Likewise, the need for inheritable
> decorators to be idempotent, in case two base classes list the same
> decorator.

Yeah, I'll add a section on "well-behaved" dynamic decorators, which
need to be aware that they may run multiple times on a single class.
If they're not naturally idempotent, they may need additional code to
be made so.

>  (For my own use attribute/method use cases, I can just have them
> remove themselves from the class's __decorators__ upon execution.)

Indeed, although that's probably unique to the approach of
programmatically modifying __decorators__. Normally, if you don't want
a decorator to be inherited and automatically applied to subclasses
you would just use an ordinary lexical decorator.

>>  You complain that metaclasses are hard to compose, and your
>> "solution" is to monkeypatch a deliberately undocumented builtin?
>
> To be clear, what I specifically proposed (as I mentioned in an earlier
> thread) was simply to patch __build_class__ in order to restore the missing
> __metaclass__ hook.  (Which, incidentally, would make ALL code using
> __metaclass__ cross-version compatible between 2.x and 3.x: a potentially
> valuable thing in and of itself!)

I did briefly consider proposing that, but then I had the dynamic
decorators idea which I like a *lot* more (since it also simplifies
other use cases that currently require a metaclass when that isn't
really what you want).

> (Automatic metaclass combining is about the only thing
> that would improve it any further.)

Automatic metaclass derivation only works in practice if the
metaclasses are all written to support cooperative multiple
inheritance though, which is a fairly large "if" (albeit, far more
likely than in the general inheritance case, since the signatures of
the methods involved in type construction are significantly more
constrained than those involved in instantiating arbitrary objects).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia