What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
On 13 October 2017 at 04:21, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
For me, the dataclasses were a typical example for inheritance, to be more precise, for metaclasses. I was astonished to see them implemented using decorators, and I was not the only one, citing Guido:
I think it would be useful to write 1-2 sentences about the problem with inheritance -- in that case you pretty much have to use a metaclass, and the use of a metaclass makes life harder for people who want to use their own metaclass (since metaclasses don't combine without some manual intervention).
Python is at a weird point here. At about every new release of Python, a new idea shows up that could be easily solved using metaclasses, yet every time we hesitate to use them, because of said necessary manual intervention for metaclass combination.
Metaclasses currently tend to serve two distinct purposes: 1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class Nobody has a problem with using metaclasses for the first purpose - that's what they're for. It's the second use case where they're problematic, as the fact that they're preserved on the class becomes a leaky implementation detail, and the lack of a JIT in CPython means they can also end up being expensive from a runtime performance perspective. Mixin classes have the same problem: something that the author may want to handle as an internal implementation detail leaks through to the runtime state of the class object. Code generating decorators like functools.total_ordering and dataclasses.dataclass (aka attr.s) instead aim at the boilerplate reduction problem directly: they let you declare in the class body the parts that you need to specify as the class designer, and then fill in at class definition time the parts that can be inferred from that base. If all you have access to is the runtime class, it behaves almost exactly as if you had written out all the autogenerated methods by hand (there may be subtle differences in the method metadata, such as the values of `__qualname__` and `__globals__`). Such decorators also do more work at class definition time in order to reduce the amount of runtime overhead introduced by reliance on chained method calls in a non-JITted Python runtime. As such, the code generating decorators have a clear domain of applicability: boilerplate reduction for class definitions without impacting the way instances behave (other than attribute and method injection), and without implicitly impacting subclass definitions (other than through regular inheritance behaviour). As far as the dataclass interaction with `__slots__` goes, that's a problem largely specific to slots (and `__metaclass__` before it), in that they're the only characteristics of a class definition that affect how CPython allocates memory for the class object itself (the descriptors for the slots are stored as a pointer array after the class struct, rather than only in the class dict). Given PEP 526 variable annotations, __slots__ could potentially benefit from a __metaclass__ style makeover, allowing an "infer_slots=True" keyword argument to type.__new__ to request that the list of slots be inferred from __annotations__ (Slot inference would conflict with setting class level default values, but that's a real conflict, as you'd be trying to use the same name on the class object for both the slot descriptor and the default value) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Metaclasses currently tend to serve two distinct purposes:
1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class
Nobody has a problem with using metaclasses for the first purpose - that's what they're for.
I am that nobody. The examples you give would be much nicer solved with decorators. Especially for ABCs it would be much better, because the fact that a class is an ABC is explicitly not inherited - its entire reason of existence is to be inherited with all the abstractness going away. And yet, currently the concrete class will still inherit from ABC. The following way of writing ABCs looks much nicer to me: @abstractclass class Spam: @abstractmethod def ham(self): ... The same holds for enums. Inheriting from enums is possible, but weird, given that you cannot add new enums to it. So, especially when comparing to the dataclasses, the following looks appealing to me: @enum class Breakfast: spam = 0 ham = 1 I'm not an expert on ORMs, but they seem to me a lot like data classes in the context of this discussion. I am aware that currently some things are easier done with metaclasses. But given that decorators can create an entirely new class if they like, they have all the power to do what they want, and even in a way much easier understood by people. As an example, here the autoslot decorator: def autoslot(cls): """turn all class variables into slots""" cls.__slots__ = list(cls.__dict__) return type(cls.__name__, cls.__bases__, class.__dict__) So I am personally more and more leaning towards a metaclass-free future. Cheers Martin
On 13 October 2017 at 19:35, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
Metaclasses currently tend to serve two distinct purposes:
1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class
Nobody has a problem with using metaclasses for the first purpose - that's what they're for.
I am that nobody. The examples you give would be much nicer solved with decorators. Especially for ABCs it would be much better, because the fact that a class is an ABC is explicitly not inherited - its entire reason of existence is to be inherited with all the abstractness going away. And yet, currently the concrete class will still inherit from ABC.
Aye, ABCs are definitely a case where I think it would be valuable to have a class decorator that: 1. Transplants any concrete method implementations from the ABC 2. Ensures that the class being defined actually implements all the required abstract methods The register method doesn't do either of those things, while inheritance has the unwanted side-effect of changing the metaclass of even concrete subclasses. As a handwavey concept, something like: @abc.implements(collections.Mapping) class MyMapping: ... # Just implement the abstract methods, get the rest injected So I am personally more and more leaning towards a metaclass-free future.
I agree with this view in the sense that I'd like the number of use cases that *require* a custom metaclass to get ever smaller (replacing them with regular inheritance + definition time method injection), but I also think they'll always have a place as a way for folks to explore the design space of what's possible given full control over the class definition process. That way, proposals like __init_subclass__ and __set_name__ can be based on pattern extraction from cases where people have decided the feature was valuable enough to be worth the hassle of maintaining a custom metaclass. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This is food for thought. I'll have to let it sink in a bit, but you may be on to something. Since the question was asked at some point, yes, metaclasses are much older than class decorators. At some point I found the book Putting Metaclasses to Work by Ira Forman and Scott Danforth ( https://www.amazon.com/Putting-Metaclasses-Work-Ira-Forman/dp/0201433052) and translated the book's ideas from C++ to Python, except for the automatic merging of multiple inherited metaclasses. But in many cases class decorators are more useful. I do worry that things like your autoslots decorator example might be problematic because they create a new class, throwing away a lot of work that was already done. But perhaps the right way to address this would be to move the decision about the instance layout to a later phase? (Not sure if that makes sense though.) --Guido On Fri, Oct 13, 2017 at 2:35 AM, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
Metaclasses currently tend to serve two distinct purposes:
1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class
Nobody has a problem with using metaclasses for the first purpose - that's what they're for.
I am that nobody. The examples you give would be much nicer solved with decorators. Especially for ABCs it would be much better, because the fact that a class is an ABC is explicitly not inherited - its entire reason of existence is to be inherited with all the abstractness going away. And yet, currently the concrete class will still inherit from ABC. The following way of writing ABCs looks much nicer to me:
@abstractclass class Spam: @abstractmethod def ham(self): ...
The same holds for enums. Inheriting from enums is possible, but weird, given that you cannot add new enums to it. So, especially when comparing to the dataclasses, the following looks appealing to me:
@enum class Breakfast: spam = 0 ham = 1
I'm not an expert on ORMs, but they seem to me a lot like data classes in the context of this discussion.
I am aware that currently some things are easier done with metaclasses. But given that decorators can create an entirely new class if they like, they have all the power to do what they want, and even in a way much easier understood by people.
As an example, here the autoslot decorator:
def autoslot(cls): """turn all class variables into slots""" cls.__slots__ = list(cls.__dict__) return type(cls.__name__, cls.__bases__, class.__dict__)
So I am personally more and more leaning towards a metaclass-free future.
Cheers
Martin _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On 10/13/2017 02:35 AM, Martin Teichmann wrote:
Metaclasses currently tend to serve two distinct purposes:
1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class
Nobody has a problem with using metaclasses for the first purpose - that's what they're for.
I am that nobody. The examples you give would be much nicer solved with decorators.
The same holds for enums. Inheriting from enums is possible, but weird, given that you cannot add new enums to it. So, especially when comparing to the dataclasses, the following looks appealing to me:
@enum class Breakfast: spam = 0 ham = 1
Things that will not work if Enum does not have a metaclass: list(EnumClass) -> list of enum members dir(EnumClass) -> custom list of "interesting" items len(EnumClass) -> number of members member in EnumClass -> True or False - protection from adding, deleting, and changing members - guards against reusing the same name twice - possible to have properties and members with the same name (i.e. "value" and "name") -- ~Ethan~
Things that will not work if Enum does not have a metaclass:
list(EnumClass) -> list of enum members dir(EnumClass) -> custom list of "interesting" items len(EnumClass) -> number of members member in EnumClass -> True or False
- protection from adding, deleting, and changing members - guards against reusing the same name twice - possible to have properties and members with the same name (i.e. "value" and "name")
In current Python this is true. But if we would go down the route of PEP 560 (which I just found, I wasn't involved in its discussion), then we could just add all the needed functionality to classes. I would do it slightly different than proposed in PEP 560: classmethods are very similar to methods on a metaclass. They are just not called by the special method machinery. I propose that the following is possible: >>> class Spam: ... @classmethod ... def __getitem__(self, item): ... return "Ham" >>> Spam[3] Ham this should solve most of your usecases. When thinking about how an automatic metaclass combiner would look like, I realized that it should ideally just reproduce the class mro, just with metaclasses. So if a class has an mro of [A, B, C, object], its metaclass should have an mro of unique_everseen([type(A), type(B), type(C), type]). But in this case, why add this layer at all? Just give the class the ability to do everything a metaclass could do, using mechanisms like @classmethod, and we're done. Greetings Martin
On 10/14/2017 07:37 AM, Martin Teichmann wrote:
Things that will not work if Enum does not have a metaclass:
list(EnumClass) -> list of enum members dir(EnumClass) -> custom list of "interesting" items len(EnumClass) -> number of members member in EnumClass -> True or False
- protection from adding, deleting, and changing members - guards against reusing the same name twice - possible to have properties and members with the same name (i.e. "value" and "name")
In current Python this is true. But if we would go down the route of PEP 560 (which I just found, I wasn't involved in its discussion), then we could just add all the needed functionality to classes.
I would do it slightly different than proposed in PEP 560: classmethods are very similar to methods on a metaclass. They are just not called by the special method machinery. I propose that the following is possible:
>>> class Spam: ... @classmethod ... def __getitem__(self, item): ... return "Ham"
>>> Spam[3] Ham
this should solve most of your usecases.
The problem with your solution is you couldn't then have a __getitem__ for the instances -- it's an either/or situation. The problem with PEP 560 is that it doesn't allow the class definition protections that a metaclass does. -- ~Ethan~
On 14 October 2017 at 17:49, Ethan Furman <ethan@stoneleaf.us> wrote:
The problem with PEP 560 is that it doesn't allow the class definition protections that a metaclass does.
Since the discussion turned to PEP 560, I can say that I don't want this to be a general mechanism, the PEP rationale section gives several specific examples why we don't want metaclasses to implement generic class machinery/internals. Could you please elaborate more what is wrong with PEP 560 and what do you mean by "class definition protections" -- Ivan
On 10/14/2017 08:57 AM, Ivan Levkivskyi wrote:
On 14 October 2017 at 17:49, Ethan Furman wrote:
The problem with PEP 560 is that it doesn't allow the class definition protections that a metaclass does.
Since the discussion turned to PEP 560, I can say that I don't want this to be a general mechanism, the PEP rationale section gives several specific examples why we don't want metaclasses to implement generic class machinery/internals.
Could you please elaborate more what is wrong with PEP 560 and what do you mean by "class definition protections"
Nothing is wrong with PEP 560. What I am referring to is: class MyEnum(Enum): red = 0 red = 1 The Enum metaclass machinery will raise an error at the "red = 1" line because it detects the redefinition of "red". This check can only happen during class definition, so only the metaclass can do it. -- ~Ethan~
On 15 October 2017 at 02:14, Ethan Furman <ethan@stoneleaf.us> wrote:
On 10/14/2017 08:57 AM, Ivan Levkivskyi wrote:
On 14 October 2017 at 17:49, Ethan Furman wrote:
The problem with PEP 560 is that it doesn't allow the class definition
protections that a metaclass does.
Since the discussion turned to PEP 560, I can say that I don't want this
to be a general mechanism, the PEP rationale section gives several
specific
examples why we don't want metaclasses to implement generic class machinery/internals.
Could you please elaborate more what is wrong with PEP 560 and what do you
mean by "class definition protections"
Nothing is wrong with PEP 560. What I am referring to is:
class MyEnum(Enum): red = 0 red = 1
The Enum metaclass machinery will raise an error at the "red = 1" line because it detects the redefinition of "red". This check can only happen during class definition, so only the metaclass can do it.
That's not necessarily an inherent restriction though - if we did decide to go even further in the direction of "How do we let base classes override semantics that currently require a custom metaclass?", then there's a fairly clear parallel between "mcl.__init__/bases.__init_subclass__" and "mcl.__prepare__/bases.__prepare_subclass__". OTOH, if you have multiple bases with competing __prepare__ methods you really *should* get a metaclass conflict, since the class body can only be executed in one namespace. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 14 October 2017 at 18:14, Ethan Furman <ethan@stoneleaf.us> wrote:
On 10/14/2017 08:57 AM, Ivan Levkivskyi wrote:
Could you please elaborate more what is wrong with PEP 560 and what do you mean by "class definition protections"
Nothing is wrong with PEP 560. What I am referring to is:
[snip]
OK thanks, then let us keep PEP 560 to its original scope. Its design is specific to generic classes, so it will probably not help with "wider" metaclass problems. As a side note, I don't think elimination of metaclasses should be a "goal by itself". This is a powerful and flexible mechanism, but there are specific situations where metaclasses don't work well because of e.g. frequent conflicts or performance penalties. -- Ivan
On 10/14/2017 11:30 AM, Ivan Levkivskyi wrote:
As a side note, I don't think elimination of metaclasses should be a "goal by itself". This is a powerful and flexible mechanism, but there are specific situations where metaclasses don't work well because of e.g. frequent conflicts or performance penalties.
+1 -- ~Ethan~
On 14 Oct 2017, at 16:37, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
Things that will not work if Enum does not have a metaclass:
list(EnumClass) -> list of enum members dir(EnumClass) -> custom list of "interesting" items len(EnumClass) -> number of members member in EnumClass -> True or False
- protection from adding, deleting, and changing members - guards against reusing the same name twice - possible to have properties and members with the same name (i.e. "value" and "name")
In current Python this is true. But if we would go down the route of PEP 560 (which I just found, I wasn't involved in its discussion), then we could just add all the needed functionality to classes.
I would do it slightly different than proposed in PEP 560: classmethods are very similar to methods on a metaclass. They are just not called by the special method machinery. I propose that the following is possible:
class Spam: ... @classmethod ... def __getitem__(self, item): ... return "Ham"
Spam[3] Ham
this should solve most of your usecases.
Except when you want to implement __getitem__ for instances as well :-). An important difference between @classmethod and methods on the metaclass is that @classmethod methods live in the same namespace as instance methods, while methods on the metaclass don’t. I ran into similar problems in PyObjC: Apple’s Cocoa libraries use instance and class methods with the same name. That when using methods on a metaclass, but not when using something similar to @classmethod. Because of this PyObjC is a heavy user of metaclasses (generated from C for additional fun). A major disadvantage of this is that tends to confuse smart editors. Ronald
While I really can't continue to be active in this discussion now [*], here are some thoughts based on observations I made: These three PEPs are all essentially solving an occurrence of the same problem: PEP 549 Instance descriptors PEP 562 Module __getattr__ PEP 560 Core support for generic types (the __class_getitem__ part) https://www.python.org/dev/peps/pep-0549/ https://www.python.org/dev/peps/pep-0562/ https://www.python.org/dev/peps/pep-0560/ PEPs 549 and 562 want an instance of ModuleType (i.e. modules) to define something on itself that looks like there was something defined on the class. For PEP 549 this is a property and for 562 it's a __getattr__ method. PEP 560 wants a __class_getitem__ method, defined on a class (instance of a metaclass), to look like there was a __getitem__ on the metaclass. PEP 560 is thus an attempt at a more fine-grained definition of a metaclass-like feature, where conflicts are less likely or can potentially be better dealth with. While PEPs 549 and 562 are doing a very similar thing to PEP 560 in theory, these use cases do not fall nicely into Nick's classification of uses for metaclasses in the email below. PEP 560 is trying to avoid a metaclass (a subclass of type) as an additional base class specifically for one class object). PEPs 549 and 562 are trying to avoid an additional class (a subclass of ModuleType) as an additional base class specifically for this one module. Whether or not fine-grainedness is the answer, it might make sense to list more different related use cases. Probably even the peps repo has more examples than the three I listed above. ––Koos [*] I'll try to be able to do what's nee½ded for the PEP 555 discussion — no1/3w still on python-ideas. On Fri, Oct 13, 2017 at 9:30 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 13 October 2017 at 04:21, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
For me, the dataclasses were a typical example for inheritance, to be more precise, for metaclasses. I was astonished to see them implemented using decorators, and I was not the only one, citing Guido:
I think it would be useful to write 1-2 sentences about the problem with inheritance -- in that case you pretty much have to use a metaclass, and the use of a metaclass makes life harder for people who want to use their own metaclass (since metaclasses don't combine without some manual intervention).
Python is at a weird point here. At about every new release of Python, a new idea shows up that could be easily solved using metaclasses, yet every time we hesitate to use them, because of said necessary manual intervention for metaclass combination.
Metaclasses currently tend to serve two distinct purposes:
1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class
Nobody has a problem with using metaclasses for the first purpose - that's what they're for.
It's the second use case where they're problematic, as the fact that they're preserved on the class becomes a leaky implementation detail, and the lack of a JIT in CPython means they can also end up being expensive from a runtime performance perspective.
Mixin classes have the same problem: something that the author may want to handle as an internal implementation detail leaks through to the runtime state of the class object.
Code generating decorators like functools.total_ordering and dataclasses.dataclass (aka attr.s) instead aim at the boilerplate reduction problem directly: they let you declare in the class body the parts that you need to specify as the class designer, and then fill in at class definition time the parts that can be inferred from that base.
If all you have access to is the runtime class, it behaves almost exactly as if you had written out all the autogenerated methods by hand (there may be subtle differences in the method metadata, such as the values of `__qualname__` and `__globals__`).
Such decorators also do more work at class definition time in order to reduce the amount of runtime overhead introduced by reliance on chained method calls in a non-JITted Python runtime.
As such, the code generating decorators have a clear domain of applicability: boilerplate reduction for class definitions without impacting the way instances behave (other than attribute and method injection), and without implicitly impacting subclass definitions (other than through regular inheritance behaviour).
As far as the dataclass interaction with `__slots__` goes, that's a problem largely specific to slots (and `__metaclass__` before it), in that they're the only characteristics of a class definition that affect how CPython allocates memory for the class object itself (the descriptors for the slots are stored as a pointer array after the class struct, rather than only in the class dict).
Given PEP 526 variable annotations, __slots__ could potentially benefit from a __metaclass__ style makeover, allowing an "infer_slots=True" keyword argument to type.__new__ to request that the list of slots be inferred from __annotations__ (Slot inference would conflict with setting class level default values, but that's a real conflict, as you'd be trying to use the same name on the class object for both the slot descriptor and the default value)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ k7hoven%40gmail.com
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Fri, Oct 13, 2017, at 02:30, Nick Coghlan wrote:
Metaclasses currently tend to serve two distinct purposes:
1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class
Nobody has a problem with using metaclasses for the first purpose - that's what they're for.
It's the second use case where they're problematic, as the fact that they're preserved on the class becomes a leaky implementation detail, and the lack of a JIT in CPython means they can also end up being expensive from a runtime performance perspective.
What about a metaclass that isn't a metaclass? A metaclass can be any callable and can return any object, e.g. a normal type. def AutoSlotMeta(name, bases, dct, real_metaclass=type): """turn all class variables into slots""" dct['__slots__'] = list(dct) return real_metaclass(name, bases, dct)
participants (8)
-
Ethan Furman -
Guido van Rossum -
Ivan Levkivskyi -
Koos Zevenhoven -
Martin Teichmann -
Nick Coghlan -
Random832 -
Ronald Oussoren