[Python-ideas] A (meta)class algebra

[Nick Coghlan]

On 14 February 2015 at 03:27, Antony Lee <antony.lee at berkeley.edu> wrote:
> While PEP422 is somewhat useful for heavy users of metaclasses (actually not
> necessarily so heavy; as mentioned in the thread you'll hit the problem as
> soon as you want to make e.g. a QObject also instance of another metaclass),
> I don't like it too much because it feels really ad hoc (adding yet another
> layer to the already complex class initialization machinery), whereas
> dynamically creating the sub-metaclass (inheriting from all the required
> classes) seems to be the "right" solution.

The need to better explain the rationale for the design was one of the
pieces of feedback from the previous review.

__init_class__ is a pattern extraction refactoring (one first applied
successfully by the Zope project more than 10 years ago). A fairly
common usage model for metaclasses is to provide class creation time
behaviour that gets inherited by subclasses, but otherwise has no
effect on the runtime behaviour of the class or subclasses. Currently,
the only way to obtain that is through a custom metaclass, which may
also impact other class behaviours, and hence they don't combine
readily, and you can't add one to an existing public class without
breaking backwards compatibility.

That pattern is common enough to potentially be worth pulling out into
a mechanism that *can't* have a runtime impact on class object
behaviour, expecially since an existing class could adopt such a
feature while remaining backwards compatible with subclasses that add
a custom metaclass.

While this does add a second mechanism to achieve something that a
custom metaclass can already do, developers are still left with a
relatively straightforward design decision: if you need to impact the
runtime behaviour of the class or its subclasses (the way abc.ABCMeta
and enum.EnumMeta do), then you still need a custom metaclass. If you
only need inherited class definition time behaviour, and only need to
support versions of Python that support __init_class__, then you
should use it is instead, as its a much simpler mechanism for people
to understand, and hence presents much lower barriers to future
maintainability.

That kind of "if the new, more specific, mechanism applies, you should
use it, as it will be easier to write, read and maintain" situation is
the result you're looking for any time you're attempting to do pattern
extraction like this.

The key problem with custom metaclasses is that once they're in the
mix, all bets regarding type system behaviour are off - you can use
those to make classes do pretty much anything you want (the abc
module, enum and SQL ORMs are readily available examples, while
systems like Zope and PEAK show just how much customisation of the
type system can already be done just through the application of custom
metaclasses).

Asking people to learn both how metaclasses in general work *and then*
how a  new automatic metaclass combination works before they can
maintain your code effectively doesn't remove any barriers to project
maintainability - instead, it adds a new one.

> In a sense, using an explicit
> and ad hoc mixer ("__add__") would be like saying that every class should
> know when it is part of a multiple inheritance chain, whereas the implicit
> ("multi_meta") mixer just relies on Python's MRO to do the right thing.  Of
> course, best would be that the "multi_meta" step be directly implemented at
> the language level, and I agree that having to add it manually isn't too
> beautiful.

Custom metaclasses are inherently hard to understand. When type is a
subclass of object, while object is itself an instance of type, and
when using a custom metaclass lets you completely rewrite the rules
for class creation, instance creation, attribute lookup, subclassing
checks, instance checks, and more, learning how to wield them
effectively does give you great power, but it also means recognising
that every time you decide to use one you're significantly reducing
the number of people that are going to be able to effectively
contribute to and help maintain that specific part of your project.
When you do decide its worthwhile to introduce one, it's generally
because the additional complexity in the class hierarchy
implementation pays for itself elsewhere in code that is more commonly
modified (for example, compare the number of ABCs, enumerations and
ORM table definitions to the number of ABC implementations,
enumeration implementations and ORM implementations).

The key goal of PEP 422 is to take a significant number of use cases
that currently demand the use of a custom metaclass and instead make
it possible to implement, understand and maintain them with roughly
the same level of knowledge of the inner workings of Python's runtime
type system as is needed to implement a custom __new__ method.

> __init_class__ could then simply be implemented as part of a normal
> metaclass, from which you could inherit without worrying about other
> metaclasses.

__init_class__ *does* get implemented as part of a metaclass: type.
type anchors the standard library's metaclass hierarchy (and most
other metaclass hierarchys) the way object anchors the overall object
hierarchy.

Once you grasp how type can be a subclass of object, while object is
itself an instance of type, then you've tackled pretty much the most
brainbending part of how the runtime type system bootstraps itself :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia