A way out of Meta-hell (was: A (meta)class algebra)
Hi everyone,
there seems to be a general agreement that metaclasses, in the current state, are very problematic. As to what the actual problem is there seems to be no agreement, but that there IS a problem seems to be accepted.
Where do we go from here? One great idea is PEP 422. It just replaces the idea of metaclasses with something simpler. One point of critique is that it will take ages until people can actually use it. In order to avoid this, I wrote a pure python implementation of PEP 422 that works all the way down to python 2.7. If everyone simply started using this, we could have a rather smooth transition.
I think that implementing PEP 422 as part of the language only makes sense if we once would be able to drop metaclasses altogether. I thought about adding a __new_class__ to PEP 422 that would simulate the __new__ in metaclasses, thinking that this is the only way metaclasses are used.
I was wrong. Looking at PyQt (to be precise: sip), I realized that it uses more: it overwrites get/setattr. That's actually a good usecase that cannot be replaced by PEP 422. (Sadly, in the case of sip, this is technically not a good usecase: it is apparently used to give the possibility to write mixin classes to the right of the mixed-in class in the inheritance chain. Could someone please convince Phil of PyQt that mixins go to the left of other base classes?)
The upshot is: my solution sketched above simply does not work for C extensions that use metaclasses.
This brings me to a completely different option: why don't we implement PEP 422 in pure python, and put it in the standard library? Then everyone writing some simple class initializer would just use that standard metaclass, and so you can use multiple inheritance and initializers, as all bases will use the same standard metaclass. Someone writing more complicated metaclass would inherit from said standard metaclass, if it makes sense to also have class initializers. For C extensions: is it possible as a C metaclass to inherit from a python base class?
I added my pure python implementation of PEP 422. It is written to be backwards compatible, so a standard library version could be simplified. A class wanting to have initializers should simply inherit from WithInit in my code.
Greetings
Martin
class Meta(type): @classmethod def __prepare__(cls, name, bases, namespace=None, **kwds): if namespace is not None: cls.__namespace__ = namespace if hasattr(cls, '__namespace__'): return cls.__namespace__() else: return super().__prepare__(name, bases, **kwds)
def __new__(cls, name, bases, dict, **kwds): if '__init_class__' in dict: dict['__init_class__'] = classmethod(dict['__init_class__']) return super(Meta, cls).__new__(cls, name, bases, dict)
def __init__(self, name, bases, dict, **kwds): self.__init_class__(**kwds)
def __init_class__(cls, **kwds): pass
WithInit = Meta("WithInit", (object,), dict(__init_class__=__init_class__))
# black magic: assure everyone is using the same WithInit import sys if hasattr(sys, "WithInit"): WithInit = sys.WithInit else: sys.WithInit = WithInit
On Sat, Feb 14, 2015 at 10:34 AM, Martin Teichmann lkb.teichmann@gmail.com wrote:
Hi everyone,
there seems to be a general agreement that metaclasses, in the current state, are very problematic. As to what the actual problem is there seems to be no agreement, but that there IS a problem seems to be accepted.
Where do we go from here? One great idea is PEP 422. It just replaces the idea of metaclasses with something simpler. One point of critique is that it will take ages until people can actually use it. In order to avoid this, I wrote a pure python implementation of PEP 422 that works all the way down to python 2.7. If everyone simply started using this, we could have a rather smooth transition.
"Everyone" here means authors of every library that uses metaclasses, right? I think you'll have a better chance convincing Python developers. They're a subset :)
I think that implementing PEP 422 as part of the language only makes sense if we once would be able to drop metaclasses altogether. I thought about adding a __new_class__ to PEP 422 that would simulate the __new__ in metaclasses, thinking that this is the only way metaclasses are used.
Well, if you want Python to drop metaclasses, the way starts with PEP 422. You have to introduce the alternative first, and then wait a *really* long time until you drop a feature. I think __new_class__ can be added after PEP 422 is in, if it turns out to be necessary.
I was wrong. Looking at PyQt (to be precise: sip), I realized that it uses more: it overwrites get/setattr. That's actually a good usecase that cannot be replaced by PEP 422. (Sadly, in the case of sip, this is technically not a good usecase: it is apparently used to give the possibility to write mixin classes to the right of the mixed-in class in the inheritance chain. Could someone please convince Phil of PyQt that mixins go to the left of other base classes?)
The upshot is: my solution sketched above simply does not work for C extensions that use metaclasses.
This brings me to a completely different option: why don't we implement PEP 422 in pure python, and put it in the standard library? Then everyone writing some simple class initializer would just use that standard metaclass, and so you can use multiple inheritance and initializers, as all bases will use the same standard metaclass. Someone writing more complicated metaclass would inherit from said standard metaclass, if it makes sense to also have class initializers.
Just putting it in the standard library doesn't make sense, since it would still only be available there from Python 3.5 on (or whenever it gets in). It really makes more sense to put this into the *real* standard metaclass (i.e. `type`). The Python implementation can be on PyPI for projects needing the backwards compatibility.
For C extensions: is it possible as a C metaclass to inherit from a python base class?
Not really, but you can rewrite the metaclass as a C extension (maybe after the Python variant is ironed out).
I added my pure python implementation of PEP 422. It is written to be backwards compatible, so a standard library version could be simplified. A class wanting to have initializers should simply inherit from WithInit in my code.
Greetings
Martin
class Meta(type): @classmethod def __prepare__(cls, name, bases, namespace=None, **kwds): if namespace is not None: cls.__namespace__ = namespace if hasattr(cls, '__namespace__'): return cls.__namespace__() else: return super().__prepare__(name, bases, **kwds)
def __new__(cls, name, bases, dict, **kwds): if '__init_class__' in dict: dict['__init_class__'] = classmethod(dict['__init_class__']) return super(Meta, cls).__new__(cls, name, bases, dict) def __init__(self, name, bases, dict, **kwds): self.__init_class__(**kwds)def __init_class__(cls, **kwds): pass
WithInit = Meta("WithInit", (object,), dict(__init_class__=__init_class__))
# black magic: assure everyone is using the same WithInit import sys if hasattr(sys, "WithInit"): WithInit = sys.WithInit else: sys.WithInit = WithInit
I haven't studied the PEP in detail, but I'm not sure this adheres to it; see https://www.python.org/dev/peps/pep-0422/#calling-init-class-from-type-init I suggest adapting the tests from http://bugs.python.org/issue17044 for this.
Hi Petr, Hi all,
Just putting it in the standard library doesn't make sense, since it would still only be available there from Python 3.5 on (or whenever it gets in). It really makes more sense to put this into the *real* standard metaclass (i.e. `type`). The Python implementation can be on PyPI for projects needing the backwards compatibility.
That was what I was thinking about. Put the python implementation on PyPI, so everyone can use it, and finally put the thing in the standard library.
I actually prefer the standard library over adding it to type, it's much less a hazzle. Given that there are use cases for metaclasses not covered by PEP 422, I guess they won't ever be dropped, so having two complicated things in C are not such a great idea in my opinion if we can easily write one of them in Python.
Not really, but you can rewrite the metaclass as a C extension (maybe after the Python variant is ironed out).
Well said, and yes, it needs ironing out. I just realized it doesn't work since I'm calling __init_class__ before its __class__ is set. But this is a solvable problem: Let's rewrite PEP 422 so that the initialization is done on subclasses, not the class itself. This is actually a very important usecase anyways, and if you need your method to be called on yourself, just call it after the class definition! One line of code should not be such a big deal.
I also simplified the code, requiring now that decorates __init_subclass__ with @classmethod.
Greetings
Martin
class Meta(type): @classmethod def __prepare__(cls, name, bases, namespace=None, **kwargs): if namespace is not None: cls.__namespace__ = namespace if hasattr(cls, '__namespace__'): return cls.__namespace__() else: return super().__prepare__(name, bases, **kwargs)
def __new__(cls, name, bases, dict, **kwargs): return super(Meta, cls).__new__(cls, name, bases, dict)
def __init__(self, name, bases, dict, namespace=None, **kwargs): super(self, self).__init_subclass__(**kwargs)
class Base(object): @classmethod def __init_subclass__(cls, **kwargs): pass
SubclassInit = Meta("SubclassInit", (Base,), {})
On Sat, Feb 14, 2015 at 12:18 PM, Martin Teichmann lkb.teichmann@gmail.com wrote:
Hi Petr, Hi all,
Just putting it in the standard library doesn't make sense, since it would still only be available there from Python 3.5 on (or whenever it gets in). It really makes more sense to put this into the *real* standard metaclass (i.e. `type`). The Python implementation can be on PyPI for projects needing the backwards compatibility.
That was what I was thinking about. Put the python implementation on PyPI, so everyone can use it, and finally put the thing in the standard library.
I actually prefer the standard library over adding it to type, it's much less a hazzle. Given that there are use cases for metaclasses not covered by PEP 422, I guess they won't ever be dropped, so having two complicated things in C are not such a great idea in my opinion if we can easily write one of them in Python.
No. For anyone to take this seriously, it needs to be what `type` itself does. The PyPI library then can just do "WithInit = object" on new Python versions. But that's beside the point right now; if you can write this then it can be integrated either way.
Also, I'm now not sure how easily you can write this in Python, since you're already starting to take shortcuts below.
Not really, but you can rewrite the metaclass as a C extension (maybe after the Python variant is ironed out).
Well said, and yes, it needs ironing out. I just realized it doesn't work since I'm calling __init_class__ before its __class__ is set. But this is a solvable problem: Let's rewrite PEP 422 so that the initialization is done on subclasses, not the class itself. This is actually a very important usecase anyways, and if you need your method to be called on yourself, just call it after the class definition! One line of code should not be such a big deal.
-1, that would be quite surprising behavior. If doing the right thing is really not possible in Python, then I guess it does need to go to the C level.
I also simplified the code, requiring now that decorates __init_subclass__ with @classmethod.
You're writing library code. A bit more complexity here i worth it. See again the Rejected Design Options in the PEP.
class Meta(type): @classmethod def __prepare__(cls, name, bases, namespace=None, **kwargs): if namespace is not None: cls.__namespace__ = namespace if hasattr(cls, '__namespace__'): return cls.__namespace__() else: return super().__prepare__(name, bases, **kwargs)
def __new__(cls, name, bases, dict, **kwargs): return super(Meta, cls).__new__(cls, name, bases, dict) def __init__(self, name, bases, dict, namespace=None, **kwargs): super(self, self).__init_subclass__(**kwargs)class Base(object): @classmethod def __init_subclass__(cls, **kwargs): pass
SubclassInit = Meta("SubclassInit", (Base,), {})
I think it's time you put this in a repository; e-mailing successive versions is inefficient. Also, run the tests on it -- that might uncover more problems.
Hi Petr, Hi all,
Well said, and yes, it needs ironing out. I just realized it doesn't work since I'm calling __init_class__ before its __class__ is set. But this is a solvable problem: Let's rewrite PEP 422 so that the initialization is done on subclasses, not the class itself. This is actually a very important usecase anyways, and if you need your method to be called on yourself, just call it after the class definition! One line of code should not be such a big deal.
-1, that would be quite surprising behavior.
Honestly, the opposite would be (actually was to me) surprising behavior. One standard usage for PEP 422 are registry classes that register their subclasses. From my experience I can tell you it's just annoying having to assure not to register the registry class itself...
It also makes a lot of sense: what should __init_class__ do? Initializing the class? But that's what the class body is already doing! The point is that we want to initialize the subclasses of a class, why should we initialize the class?
I also simplified the code, requiring now that decorates __init_subclass__ with @classmethod.
You're writing library code. A bit more complexity here i worth it. See again the Rejected Design Options in the PEP.
I'm fine with or without explicit classmethod, I can put it back in if people prefer it. I had just realized that at least the test code for the currently proposed C implementation still uses classmethod.
I think it's time you put this in a repository; e-mailing successive versions is inefficient.
Done. It's at https://github.com/tecki/metaclasses
Also, run the tests on it -- that might uncover more problems.
Done. For sure I had to modify the tests so that my approach that subclasses get initialized is tested - which makes the tests ugly, but they run.
I also took out the part where other metaclasses may block __init_class__, that doesn't really make sense in an approach like mine, it's much more easy to simply circumvent my metaclass.
If it is really desired it should be no problem to put it in as well.
Greetings
Martin
On 14 February 2015 at 20:23, Petr Viktorin encukou@gmail.com wrote:
On Sat, Feb 14, 2015 at 10:34 AM, Martin Teichmann lkb.teichmann@gmail.com wrote:
I think that implementing PEP 422 as part of the language only makes sense if we once would be able to drop metaclasses altogether. I thought about adding a __new_class__ to PEP 422 that would simulate the __new__ in metaclasses, thinking that this is the only way metaclasses are used.
Well, if you want Python to drop metaclasses, the way starts with PEP 422. You have to introduce the alternative first, and then wait a *really* long time until you drop a feature. I think __new_class__ can be added after PEP 422 is in, if it turns out to be necessary.
There's nothing wrong with metaclasses per se - they work fine for implementing things like abc.ABC, enum.Enum, SQL ORMs and more. There's just a significant subset of their current use cases that doesn't need their full power and could benefit from having a simpler alternative mechanism available.
PEP 422 aims to carve out that subset and provide a simpler way of doing the same thing. A pure Python equivalent has existed in Zope for over a decade (which is referenced from the PEP after someone pointed it out in one of the earlier review rounds).
If someone had the time to update the PEP to address the issues raised in the last round of reviews, I'd be happy to add a third co-author :)
Cheers, Nick.
participants (3)
-
Martin Teichmann -
Nick Coghlan -
Petr Viktorin