Mailman 3 Possible rough edges in Python 3 metaclasses (was Re: Language reference updated for metaclasses) - Python-Dev

newer
Re: [Python-Dev] [Python-checkins]...

Possible rough edges in Python 3 metaclasses (was Re: Language reference updated for metaclasses)

older
Open PEPs and large-scale changes...

Nick Coghlan

4 Jun 2012 4 Jun '12

11:18 p.m.

On Tue, Jun 5, 2012 at 8:58 AM, PJ Eby wrote:

...

On Mon, Jun 4, 2012 at 6:15 PM, Nick Coghlan wrote:

...
It's actually the pre-decoration class, since the cell is initialised before the class is passed to the first decorator. I agree it's a little weird, but I did try to describe it accurately in the new docs.

I see that now; it might be helpful to explicitly call that out.

This is adding to my list of Python 3 metaclass gripes, though. In Python 2, I have in-the-body-of-a-class decorators implemented using metaclasses, that will no longer work because of PEP 3115...

I'm not quite following this one - do you mean they won't support __prepare__, won't play nicely with other metaclasses that implement __prepare__, or something else?

...

and if I switch to using class decorators instead, then they won't work because of PEP 3135. :-(

Yeah, 3135 has had some "interesting" consequences :P (e.g. class body level __class__ definitions are still broken at the moment if you also reference super: http://bugs.python.org/issue12370)

...

Meanwhile, mixing metaclasses is more difficult than ever, due to __prepare__, and none of these flaws can be worked around officially, because __build_class__ is an "implementation detail". I *really* want to like Python 3, but am still hoping against hope for the restoration of hooks that __metaclass__ allowed, or some alternative mechanism that would serve the same use cases.

Specifically, my main use case is method-level decorators and attribute descriptors that need to interact with a user-defined class, *without* requiring that user-defined class to either 1) redundantly decorate the class or 2) inherit from some specific base or inject a specific metaclass. I only use __metaclass__ in 2.x for this because it's the only way for code executed in a class body to gain access to the class at creation time.

The reason for wanting this to be transparent is that 1) if you forget the redundant class-decorator, mixin, or metaclass, stuff will silently not work,

Why would it silently not work? What's preventing you from having decorators that create wrapped functions that fail noisily when called, then providing a class decorator that unwraps those functions, fixes them up with the class references they need and stores the unwrapped and updated versions back on the class. You call it redundant, I call it explicit.

...

and 2) mixing bases or metaclasses has much higher coupling to the library providing the decorators or descriptors, and greatly increases the likelihood of mixing metaclasses.

So don't do that, then. Be explicit.

...

And at the moment, the only workaround I can come up with that *doesn't* involve replacing __build_class__ is abusing the system trace hook; ISTM that replacing __build_class__ is the better of those two options.

Stop trying to be implicit. Implicit magic sucks, don't add more (PEP 3135 is bad enough).

...

At this point, with the additions of types.new_class(), ISTM that every Python implementation will have to *have* a __build_class__ function or its equivalent; all that remains is the question of whether they allow *replacing* it.

types.new_class() is actually a pure Python reimplementation of the PEP 3115 algorithm. Why would it imply the need for a __build_class__ function? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Show replies by date

Ethan Furman

4 Jun 4 Jun

11:43 p.m.

Nick Coghlan wrote:

...

On Tue, Jun 5, 2012 at 8:58 AM, PJ Eby wrote:

...
The reason for wanting this to be transparent is that 1) if you forget the redundant class-decorator, mixin, or metaclass, stuff will silently not work,

Why would it silently not work? What's preventing you from having decorators that create wrapped functions that fail noisily when called, then providing a class decorator that unwraps those functions, fixes them up with the class references they need and stores the unwrapped and updated versions back on the class.

You call it redundant, I call it explicit.

The first time you specify something, it's explicit; if you have to specify the same thing a second time, it's redundant; if this was a good thing why do we say DRY so often? ~Ethan~

PJ Eby

5 Jun 5 Jun

12:10 a.m.

On Mon, Jun 4, 2012 at 7:18 PM, Nick Coghlan wrote:

...

...
On Mon, Jun 4, 2012 at 6:15 PM, Nick Coghlan wrote:

...
It's actually the pre-decoration class, since the cell is initialised before the class is passed to the first decorator. I agree it's a little weird, but I did try to describe it accurately in the new docs.

I see that now; it might be helpful to explicitly call that out.

This is adding to my list of Python 3 metaclass gripes, though. In Python 2, I have in-the-body-of-a-class decorators implemented using

On Tue, Jun 5, 2012 at 8:58 AM, PJ Eby wrote: metaclasses,

...
that will no longer work because of PEP 3115...

I'm not quite following this one - do you mean they won't support __prepare__, won't play nicely with other metaclasses that implement __prepare__, or something else?

I mean that class-level __metaclass__ is no longer supported as of PEP 3115, so I can't use that as a way to non-invasively obtain the enclosing class at class creation time. (Unfortunately, I didn't realize until relatively recently that it wasn't supported any more; the PEP itself doesn't say the functionality will be removed. Otherwise, I'd have lobbied sooner for a better migration path.)

...

...
Meanwhile, mixing metaclasses is more difficult than ever, due to __prepare__, and none of these flaws can be worked around officially, because __build_class__ is an "implementation detail". I *really* want to like Python 3, but am still hoping against hope for the restoration of hooks that __metaclass__ allowed, or some alternative mechanism that would serve the same use cases.

Specifically, my main use case is method-level decorators and attribute descriptors that need to interact with a user-defined class, *without* requiring that user-defined class to either 1) redundantly decorate the class or 2) inherit from some specific base or inject a specific metaclass. I only use __metaclass__ in 2.x for this because it's the only way for code executed in a class body to gain access to the class at creation time.

The reason for wanting this to be transparent is that 1) if you forget the redundant class-decorator, mixin, or metaclass, stuff will silently not work,

Why would it silently not work? What's preventing you from having decorators that create wrapped functions that fail noisily when called, then providing a class decorator that unwraps those functions, fixes them up with the class references they need and stores the unwrapped and updated versions back on the class.

If you are registering functions or attributes in some type of registry either at the class level or in some sort of global registry, then the function will never be called or the attribute accessed, so there is no opportunity to give an error message saying, "you should have done this". Something will simply fail to happen that should have happened. Essentially, the lack of this hook forces things to be done outside of class bodies that make more sense to be done *inside* class bodies.

...

...
library providing the decorators or descriptors, and greatly increases

and 2) mixing bases or metaclasses has much higher coupling to the the

...
likelihood of mixing metaclasses.

So don't do that, then. Be explicit.

Easy for you to say. However, I have *many* decorators and descriptors which follow this pattern, in various separately-distributed libraries. If *each* of these libraries now grows a redundant class decorator for Python 3, then any code which uses more than one in the same class will now have a big stack of decorator noise on top of it... and missing any one of them will cause silent failure. And of course, all other libraries which use these decorators and descriptors will *also* have to add this line-noise in order to port to Python 3... including the applications that depend on those libraries. And let's not forget the people who subclass or extend any of my decorators or descriptors -- they will need to tell *their* users to begin adding redundant class-level decorators, and so on.

...

...
And at the moment, the only workaround I can come up with that *doesn't* involve replacing __build_class__ is abusing the system trace hook; ISTM that replacing __build_class__ is the better of those two options.

Stop trying to be implicit. Implicit magic sucks,

The fact that a method-level decorator or descriptor might need access to its containing class when the class is created does not in any way make it *intrinsically* "magical" or "implicit". The focus here is on the decorator or descriptor: the class access is just an implementation detail, and generally doesn't involve modifying the class itself. So, please let's not start FUDding here. None of the descriptors or decorators I'm describing are implicit or magical in the least. They are documented as doing what they do, and they use perfectly legal mechanisms of Python 2 to implement those behaviors. If tomorrow a new PEP were introduced that provided an alternate way of doing this, I'd gladly use it to migrate these features forward. Implying that my use cases "suck" does not help; I could just as easily say, "redundancy sucks", and we are no further along to a solution. It might be more helpful to propose some alternate mechanism to provide the same feature to be introduced in a future Python version, like a special __decorators__ key in a class dictionary that the default type would iterate over and call with the class. Or perhaps a proposal that the default type creation code simply iterate over the contents of the class namespace and call "ob.__used_in_class__(cls, attrname)" on each found object. Or something else altogether, any of which I would happily use in place of the old __metaclass__ hook, and which would certainly be less obscure.

...

At this point, with the additions of types.new_class(), ISTM that every

...
Python implementation will have to *have* a __build_class__ function or its equivalent; all that remains is the question of whether they allow *replacing* it.

types.new_class() is actually a pure Python reimplementation of the PEP 3115 algorithm. Why would it imply the need for a __build_class__ function?

Because now every Python implementation will contain the code to do this *twice*: once inside its normal class creation machinery, and once inside of types.new_class(). (Well, unless they share code internally, of course.) IOW, saying that __build_class__ is an implementation detail of CPython doesn't quite wash: thanks to types.new_class() every Python implementation now *must* support programmatically creating a class in this way. The only detail that remains is whether it's allowed to replace the built-in equivalent to __build_class__, and whether it's documented as a standard feature ala __import__ (and its __metaclass__ predecessor). Note, though, that I've only focused on __build_class__ because it's low-hanging fruit: it could be used to work around the absence of __metaclass__ or a more-specific hook like the ones I mentioned above, *and* it's already implemented in Python 3.x. (That doesn't make it the best solution, of course, just a low-hanging one.)

Nick Coghlan

2:25 a.m.

On Tue, Jun 5, 2012 at 10:10 AM, PJ Eby wrote:

...

On Mon, Jun 4, 2012 at 7:18 PM, Nick Coghlan wrote:

...
On Tue, Jun 5, 2012 at 8:58 AM, PJ Eby wrote:

...
On Mon, Jun 4, 2012 at 6:15 PM, Nick Coghlan wrote:

...
It's actually the pre-decoration class, since the cell is initialised before the class is passed to the first decorator. I agree it's a little weird, but I did try to describe it accurately in the new docs.

I see that now; it might be helpful to explicitly call that out.

This is adding to my list of Python 3 metaclass gripes, though. In Python 2, I have in-the-body-of-a-class decorators implemented using metaclasses, that will no longer work because of PEP 3115...

I'm not quite following this one - do you mean they won't support __prepare__, won't play nicely with other metaclasses that implement __prepare__, or something else?

I mean that class-level __metaclass__ is no longer supported as of PEP 3115, so I can't use that as a way to non-invasively obtain the enclosing class at class creation time.

(Unfortunately, I didn't realize until relatively recently that it wasn't supported any more; the PEP itself doesn't say the functionality will be removed. Otherwise, I'd have lobbied sooner for a better migration path.)

As in the "def __metaclass__(name, bases, ns): return type(name, bases, ns)" functionality? You can still pass an ordinary callable as the metaclass parameter and it will behave the same as the old class level __metaclass__ definition. I personally wouldn't be averse to bringing back the old spelling for the case where __prepare__ isn't needed - you're right that it's a convenient way to do a custom callable that gets inherited by subclasses. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

PJ Eby

3:36 a.m.

On Mon, Jun 4, 2012 at 10:25 PM, Nick Coghlan wrote:

...

...
On Mon, Jun 4, 2012 at 7:18 PM, Nick Coghlan wrote:

...
On Tue, Jun 5, 2012 at 8:58 AM, PJ Eby wrote:

...
On Mon, Jun 4, 2012 at 6:15 PM, Nick Coghlan

wrote:

...
...
...
It's actually the pre-decoration class, since the cell is initialised before the class is passed to the first decorator. I agree it's a little weird, but I did try to describe it accurately in the new docs.

I see that now; it might be helpful to explicitly call that out.

This is adding to my list of Python 3 metaclass gripes, though. In Python 2, I have in-the-body-of-a-class decorators implemented using metaclasses, that will no longer work because of PEP 3115...

I'm not quite following this one - do you mean they won't support __prepare__, won't play nicely with other metaclasses that implement __prepare__, or something else?

I mean that class-level __metaclass__ is no longer supported as of PEP 3115, so I can't use that as a way to non-invasively obtain the enclosing class at class creation time.

(Unfortunately, I didn't realize until relatively recently that it wasn't supported any more; the PEP itself doesn't say the functionality will be removed. Otherwise, I'd have lobbied sooner for a better migration

On Tue, Jun 5, 2012 at 10:10 AM, PJ Eby wrote: path.)

As in the "def __metaclass__(name, bases, ns): return type(name, bases, ns)" functionality?

The part where you can do that *dynamically in the class body* from a decorator or other code, yes. You can still pass an ordinary callable as the metaclass parameter and

...

it will behave the same as the old class level __metaclass__ definition.

Which runs afoul of the requirement that users of the in-body decorator or descriptor not have to 1) make the extra declaration and 2) deal with the problem of needing multiple metaclasses. (Every framework that wants to do this sort of thing will end up having to have its own metaclass, and you won't be able to use them together without first creating a mixed metaclass.)

...

I personally wouldn't be averse to bringing back the old spelling for the case where __prepare__ isn't needed - you're right that it's a convenient way to do a custom callable that gets inherited by subclasses.

That was a nice hack, but not one I'd lose any sleep over; the translation to PEP 3115 syntax is straightforward even if less elegant. It's the *dynamic* character that's missing in 3.x. In any case, I don't use the __metaclass__ hook to actually *change* the metaclass at all; it's simply the easiest way to get at the class as soon as its built in 2.x. If there were a way to register a function to be called as soon as an enclosing class is created, I would use that instead.

Eric Snow

2:43 a.m.

On Mon, Jun 4, 2012 at 6:10 PM, PJ Eby wrote:

...

I mean that class-level __metaclass__ is no longer supported as of PEP 3115, so I can't use that as a way to non-invasively obtain the enclosing class at class creation time.

Depends on what you mean by non-invasive: * http://code.activestate.com/recipes/577813/ * http://code.activestate.com/recipes/577867/ -eric

PJ Eby

4:03 a.m.

On Mon, Jun 4, 2012 at 10:43 PM, Eric Snow wrote:

...

On Mon, Jun 4, 2012 at 6:10 PM, PJ Eby wrote:

...
I mean that class-level __metaclass__ is no longer supported as of PEP 3115, so I can't use that as a way to non-invasively obtain the enclosing class at class creation time.

Depends on what you mean by non-invasive:

Non-invasive meaning, "not requiring the user of the descriptor or decorator to make extra declarations-at-a-distance, especially ones that increase the likelihood of clashing machinery if multiple frameworks require the same functionality." ;-) That means class decorators, mixins, and explicit metaclasses don't work. The class decorator adds yak shaving, and the others lead to functional clashing. Currently, my choices for porting these libraries (and their dependents) to Python 3 are (in roughly descending order of preference): 1. Replace __builtins__.__build_class__ and hope PyPy et al follow CPython's lead, 2. Abuse sys.set_trace() and __class__ to get a callback at the right moment (because set_trace() *is* supported on other implementations in practice right now, at least for 2.x), or 3. Write a class decorator like "@py3_is_less_dynamic_than_py2", put it in a common library, and ask everybody and their dog to use it on any class whose body contains any decorator or descriptor from any of my libraries (or which someone else *derived* from any of my libraries, etc. ad nauseam), oh and good luck finding which ones all of them are, and yeah, if you miss it, stuff might not work. (Note, by the way, that this goes against the advice of not changing APIs while migrating to Python 3... and let's not forget all the documentation that has to be changed, each piece of which must explain and motivate this new and apparently-pointless decorator to the library's user.) I would prefer to have an option 4 or 5 (where there's a standard Python way to get a class creation callback from a class body), but #1 is honestly the most attractive at the moment, as I might only need to implement it in *one* library, with no changes to clients, including people who've built stuff based on that library or any of its clients, recursively. (#2 would also do it, but I was *really* hoping to get rid of that hack in Python 3.) Given these choices, I hope it's more understandable why I'd want to lobby for at least documenting __build_class__ and its replaceability, and perhaps encouraging other Python 3 implementations to offer the same feature. Given that implementing PEP 3115 and types.new_class() mean the same functionality has to be present, and given that class creation is generally not a performance-critical function, there's little reason for a sufficiently dynamic Python implementation (by which I basically mean Jython, IronPython, and PyPy) to support it. (More-static implementations don't always even support metaclasses to begin with, so they're not going to lose anything important by not supporting dynamic __build_class__.)

Nick Coghlan

7:53 a.m.

On Tue, Jun 5, 2012 at 2:03 PM, PJ Eby wrote:

...

On Mon, Jun 4, 2012 at 10:43 PM, Eric Snow wrote:

...
On Mon, Jun 4, 2012 at 6:10 PM, PJ Eby wrote:

...
I mean that class-level __metaclass__ is no longer supported as of PEP 3115, so I can't use that as a way to non-invasively obtain the enclosing class at class creation time.

Depends on what you mean by non-invasive:

Non-invasive meaning, "not requiring the user of the descriptor or decorator to make extra declarations-at-a-distance, especially ones that increase the likelihood of clashing machinery if multiple frameworks require the same functionality." ;-)

That means class decorators, mixins, and explicit metaclasses don't work. The class decorator adds yak shaving, and the others lead to functional clashing.

Currently, my choices for porting these libraries (and their dependents) to Python 3 are (in roughly descending order of preference):

1. Replace __builtins__.__build_class__ and hope PyPy et al follow CPython's lead,

Please don't try to coerce everyone else into supporting such an ugly hack by abusing an implementation detail. There's a reason types.new_class() uses a different signature. Deliberately attempting to present python-dev with a fait accompli instead of building consensus for officially adding a feature to the language is *not cool*.

...

I would prefer to have an option 4 or 5 (where there's a standard Python way to get a class creation callback from a class body), but #1 is honestly the most attractive at the moment, as I might only need to implement it in *one* library, with no changes to clients, including people who've built stuff based on that library or any of its clients, recursively. (#2 would also do it, but I was *really* hoping to get rid of that hack in Python 3.)

Please be patient and let us work out a solution that has at least some level of consensus associated with it, rather than running off and doing your own thing. As I understand it, what you currently do is, from a running decorator, walk the stack with sys._getframes() and insert a "__metaclass__" value into the class namespace. Now, one minor annoyance with current class decorators is that they're *not* inherited. This is sometimes what you want, but sometimes you would prefer to automatically decorate all subclasses as well. Currently, that means writing a custom metaclass to automatically apply the decorators. This has all the problems you have noted with composability. It seems then, that a potentially clean solution may be found by adding a *dynamic* class decoration hook. As a quick sketch of such a scheme, add the following step to the class creation process (between the current class creation process, but before the execution of lexical decorators): for mro_cls in cls.mro(): decorators = mro_cls.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls) Would such a dynamic class decoration hook meet your use case? Such a hook has use cases (specifically involving decorator inheritance) that *don't* require the use of sys._getframes(), so is far more likely to achieve the necessary level of consensus. As a specific example, the unittest module could use it to provide test parameterisation without needing a custom metaclass.

...

Given these choices, I hope it's more understandable why I'd want to lobby for at least documenting __build_class__ and its replaceability, and perhaps encouraging other Python 3 implementations to offer the same feature.

You've jumped to a hackish solution instead of taking a step back and trying to think of a potentially elegant addition to the language that is more composable and doesn't require modification of process global state. You complain that metaclasses are hard to compose, and your "solution" is to monkeypatch a deliberately undocumented builtin? Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Michael Foord

9:11 a.m.

New subject: Possible rough edges in Python 3 metaclasses (was Re: Language reference updated for metaclasses)

On 5 Jun 2012, at 08:53, Nick Coghlan wrote:

...

[snip...]

Now, one minor annoyance with current class decorators is that they're *not* inherited. This is sometimes what you want, but sometimes you would prefer to automatically decorate all subclasses as well. Currently, that means writing a custom metaclass to automatically apply the decorators. This has all the problems you have noted with composability.

It seems then, that a potentially clean solution may be found by adding a *dynamic* class decoration hook. As a quick sketch of such a scheme, add the following step to the class creation process (between the current class creation process, but before the execution of lexical decorators):

for mro_cls in cls.mro(): decorators = mro_cls.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls)

Would such a dynamic class decoration hook meet your use case? Such a hook has use cases (specifically involving decorator inheritance) that *don't* require the use of sys._getframes(), so is far more likely to achieve the necessary level of consensus.

As a specific example, the unittest module could use it to provide test parameterisation without needing a custom metaclass.

Heh, you're effectively restoring the old Python 2 metaclass machinery with a slightly-less-esoteric mechanism. That aside I really like it. Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Nick Coghlan

10:25 a.m.

On Tue, Jun 5, 2012 at 7:11 PM, Michael Foord wrote:

...

On 5 Jun 2012, at 08:53, Nick Coghlan wrote:

...
[snip...]

Now, one minor annoyance with current class decorators is that they're *not* inherited. This is sometimes what you want, but sometimes you would prefer to automatically decorate all subclasses as well. Currently, that means writing a custom metaclass to automatically apply the decorators. This has all the problems you have noted with composability.

It seems then, that a potentially clean solution may be found by adding a *dynamic* class decoration hook. As a quick sketch of such a scheme, add the following step to the class creation process (between the current class creation process, but before the execution of lexical decorators):

for mro_cls in cls.mro(): decorators = mro_cls.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls)

Would such a dynamic class decoration hook meet your use case? Such a hook has use cases (specifically involving decorator inheritance) that *don't* require the use of sys._getframes(), so is far more likely to achieve the necessary level of consensus.

As a specific example, the unittest module could use it to provide test parameterisation without needing a custom metaclass.

Heh, you're effectively restoring the old Python 2 metaclass machinery with a slightly-less-esoteric mechanism. That aside I really like it.

Yup, writing a PEP now - I'm mostly interested in the inheritance aspects, but providing PJE with a way to get the effect he wants without monkeypatching undocumented builtins at runtime and effectively forking CPython's runtime behaviour is a useful bonus. Metaclasses *do* have a problem with composition and lexical decorators don't play nicely with inheritance, but a dynamic decorator system modelled to some degree on the old __metaclass__ mechanics could fill the gap nicely. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan

12:24 p.m.

On Tue, Jun 5, 2012 at 8:25 PM, Nick Coghlan wrote:

...

On Tue, Jun 5, 2012 at 7:11 PM, Michael Foord wrote:

...
On 5 Jun 2012, at 08:53, Nick Coghlan wrote:

...
[snip...]

Now, one minor annoyance with current class decorators is that they're *not* inherited. This is sometimes what you want, but sometimes you would prefer to automatically decorate all subclasses as well. Currently, that means writing a custom metaclass to automatically apply the decorators. This has all the problems you have noted with composability.

It seems then, that a potentially clean solution may be found by adding a *dynamic* class decoration hook. As a quick sketch of such a scheme, add the following step to the class creation process (between the current class creation process, but before the execution of lexical decorators):

for mro_cls in cls.mro(): decorators = mro_cls.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls)

Would such a dynamic class decoration hook meet your use case? Such a hook has use cases (specifically involving decorator inheritance) that *don't* require the use of sys._getframes(), so is far more likely to achieve the necessary level of consensus.

As a specific example, the unittest module could use it to provide test parameterisation without needing a custom metaclass.

Heh, you're effectively restoring the old Python 2 metaclass machinery with a slightly-less-esoteric mechanism. That aside I really like it.

Yup, writing a PEP now - I'm mostly interested in the inheritance aspects, but providing PJE with a way to get the effect he wants without monkeypatching undocumented builtins at runtime and effectively forking CPython's runtime behaviour is a useful bonus.

Metaclasses *do* have a problem with composition and lexical decorators don't play nicely with inheritance, but a dynamic decorator system modelled to some degree on the old __metaclass__ mechanics could fill the gap nicely.

PEP written and posted: http://www.python.org/dev/peps/pep-0422/ More toy examples here: https://bitbucket.org/ncoghlan/misc/src/default/pep422.py Yes, it means requiring the use of a specific metaclass in 3.2 (either directly or via inheritance), but monkeypatching an undocumented builtin is going to pathological lengths just to avoid requiring that people explicitly interoperate with your particular metaclass mechanisms. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Xavier Morel

12:37 p.m.

New subject: Possible rough edges in Python 3 metaclasses (was Re: Language reference updated for metaclasses)

On 5 juin 2012, at 14:24, Nick Coghlan wrote:

...

On Tue, Jun 5, 2012 at 8:25 PM, Nick Coghlan wrote:

...
On Tue, Jun 5, 2012 at 7:11 PM, Michael Foord wrote:

...
On 5 Jun 2012, at 08:53, Nick Coghlan wrote:

...
[snip...]

Now, one minor annoyance with current class decorators is that they're *not* inherited. This is sometimes what you want, but sometimes you would prefer to automatically decorate all subclasses as well. Currently, that means writing a custom metaclass to automatically apply the decorators. This has all the problems you have noted with composability.

It seems then, that a potentially clean solution may be found by adding a *dynamic* class decoration hook. As a quick sketch of such a scheme, add the following step to the class creation process (between the current class creation process, but before the execution of lexical decorators):

for mro_cls in cls.mro(): decorators = mro_cls.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls)

Would such a dynamic class decoration hook meet your use case? Such a hook has use cases (specifically involving decorator inheritance) that *don't* require the use of sys._getframes(), so is far more likely to achieve the necessary level of consensus.

As a specific example, the unittest module could use it to provide test parameterisation without needing a custom metaclass.

Heh, you're effectively restoring the old Python 2 metaclass machinery with a slightly-less-esoteric mechanism. That aside I really like it.

Yup, writing a PEP now - I'm mostly interested in the inheritance aspects, but providing PJE with a way to get the effect he wants without monkeypatching undocumented builtins at runtime and effectively forking CPython's runtime behaviour is a useful bonus.

Metaclasses *do* have a problem with composition and lexical decorators don't play nicely with inheritance, but a dynamic decorator system modelled to some degree on the old __metaclass__ mechanics could fill the gap nicely.

PEP written and posted: http://www.python.org/dev/peps/pep-0422/ More toy examples here: https://bitbucket.org/ncoghlan/misc/src/default/pep422.py

Does it really make sense to meld decorators and inheritance so? With this, class "decorators" become way more than mere decorators and start straying quite far away from their function-based counterparts (which are not "inherited" when overriding methods but are statically applied instead)

Nick Coghlan

1:17 p.m.

On Tue, Jun 5, 2012 at 10:37 PM, Xavier Morel wrote:

...

On 5 juin 2012, at 14:24, Nick Coghlan wrote:

...
...
Metaclasses *do* have a problem with composition and lexical decorators don't play nicely with inheritance, but a dynamic decorator system modelled to some degree on the old __metaclass__ mechanics could fill the gap nicely.

PEP written and posted: http://www.python.org/dev/peps/pep-0422/ More toy examples here: https://bitbucket.org/ncoghlan/misc/src/default/pep422.py

Does it really make sense to meld decorators and inheritance so? With this, class "decorators" become way more than mere decorators and start straying quite far away from their function-based counterparts (which are not "inherited" when overriding methods but are statically applied instead)

Lexical class decorators won't go away, and will still only apply to the associated class definition. There are a couple of key points about class decorators that make them a *lot* easier to reason about than metaclasses: 1. They have a very simple expected API: "class in, class out" 2. As a result of 1, they can typically be pipelined easily: in the absence of decorator abuse, you can take the output of one class decorator and feed it to the input of the next one Metaclasses live at a different level of complexity all together - in order to make use of them, you need to have some understanding of how a name, a sequence of bases and a namespace can be combined to create a class object. Further, because it's a transformative API (name, bases, ns -> class object), combining them requires a lot more care (generally involving inheriting from type and making appropriate use of super() calls). However, even when all you really want is to decorate the class after it has been created, defining a metaclass is currently still necessary if you also want to be notified when new subclasses are defined. This PEP proposes that those two characteristics be split: if all you want is inheritance of decorators, then dynamic decorators are a much simpler, more multiple inheritance friendly solution than using a custom metaclass, leave full metaclass definitions for cases where you really *do* want almost complete control over the class creation process. That said, actively discouraging PJE from his current scheme that involves monkeypatching __build_class__ and publishing "Python 3 (with monkeypatched undocumented builtins) compatible" packages on PyPI is definitely a key motivation in putting this proposal together. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Joao S. O. Bueno

4 p.m.

On 5 June 2012 09:24, Nick Coghlan wrote:

...

PEP written and posted: http://www.python.org/dev/peps/pep-0422/ More toy examples here: https://bitbucket.org/ncoghlan/misc/src/default/pep422.py

Yes, it means requiring the use of a specific metaclass in 3.2 (either directly or via inheritance), but monkeypatching an undocumented builtin is going to pathological lengths just to avoid requiring that people explicitly interoperate with your particular metaclass mechanisms.

When reading the PEP, I got the impression that having a "__decorate__" method on "type", which would perform its thing, would be not add magic exceptions, would be more explicit and more flexible than having an extra step to be called between the metaclass execution and decorator applying. So, I think that settling for having the decorators applied - as described in the PEP - in a __decorate__ method of the metaclass would be nicer and cleaner. js -><-

...

Regards, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br

Nick Coghlan

6 Jun 6 Jun

12:14 a.m.

On Wed, Jun 6, 2012 at 2:00 AM, Joao S. O. Bueno wrote:

...

On 5 June 2012 09:24, Nick Coghlan wrote:

...
PEP written and posted: http://www.python.org/dev/peps/pep-0422/ More toy examples here: https://bitbucket.org/ncoghlan/misc/src/default/pep422.py

Yes, it means requiring the use of a specific metaclass in 3.2 (either directly or via inheritance), but monkeypatching an undocumented builtin is going to pathological lengths just to avoid requiring that people explicitly interoperate with your particular metaclass mechanisms.

When reading the PEP, I got the impression that having a "__decorate__" method on "type", which would perform its thing, would be not add magic exceptions, would be more explicit and more flexible than having an extra step to be called between the metaclass execution and decorator applying.

So, I think that settling for having the decorators applied - as described in the PEP - in a __decorate__ method of the metaclass would be nicer and cleaner.

On reflection, I'm actually inclined to agree. The next version of the PEP will propose the addition of type.__decorate__(). This new method will be invoked *after* the class is created and the __class__ cell is populated, but *before* lexical decorators are applied. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

PJ Eby

3:48 a.m.

On Tue, Jun 5, 2012 at 8:14 PM, Nick Coghlan wrote:

...

On reflection, I'm actually inclined to agree. The next version of the PEP will propose the addition of type.__decorate__(). This new method will be invoked *after* the class is created and the __class__ cell is populated, but *before* lexical decorators are applied.

Why not have type.__call__() do the invocation? That is, why does it need to become part of the external class-building protocol? (One advantage to putting it all in type.__call__() is that you can emulate the protocol in older Pythons more easily than if it's part of the external class creation dance.)

Nick Coghlan

6:54 a.m.

On Wed, Jun 6, 2012 at 1:48 PM, PJ Eby wrote:

...

On Tue, Jun 5, 2012 at 8:14 PM, Nick Coghlan wrote:

...
On reflection, I'm actually inclined to agree. The next version of the PEP will propose the addition of type.__decorate__(). This new method will be invoked *after* the class is created and the __class__ cell is populated, but *before* lexical decorators are applied.

Why not have type.__call__() do the invocation? That is, why does it need to become part of the external class-building protocol?

(One advantage to putting it all in type.__call__() is that you can emulate the protocol in older Pythons more easily than if it's part of the external class creation dance.)

That's something else I need to write up in the PEP (I *had* thought about it, I just forgot to include the explanation for why it doesn't work). The problems are two-fold: 1. It doesn't play nicely with __class__ (since the cell doesn't get populated until after type.__call__ returns) 2. It doesn't play nicely with metaclass inheritance (when you call up to type.__call__ from a subclass __call__ implementation, the dynamic decorators will see a partially initialised class object) That's really two aspects of the same underlying concern though: the idea is that decorators should only be handed a fully initialised class instance, which means they have to be invoked *after* the metaclass invocation has already returned. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

PJ Eby

5 Jun 5 Jun

3:28 p.m.

On Tue, Jun 5, 2012 at 3:53 AM, Nick Coghlan wrote:

...

Please don't try to coerce everyone else into supporting such an ugly hack by abusing an implementation detail.

Whoa, whoa there. Again with the FUD. Sorry if I gave the impression that I'm about to unleash the monkeypatching hordes tomorrow or something. I'm unlikely to begin serious Python 3 porting of the relevant libraries before 3.3 is released; the reason I'm talking about this now is because there's currently Python-Dev discussion regarding metaclasses, so it seemed like a good time to bring the subject up, to see if there were any *good* solutions. Just because my three existing options are, well, the only options if I started porting today, doesn't mean I think they're the *only* options. As I said, an option 4 or 5 would be fantastic, and your new PEP 422 is wonderful in that regard. Thank you VERY much for putting that together, I appreciate that very much. (I just wish you hadn't felt forced or coerced to do it -- that was not my intention *at all*!) Frankly, a big part of my leaning towards the __build_class__ option was that it's the *least work by Python implementors* (at least in terms of coding) to get my use case met. The reason I didn't write a PEP myself is because I didn't want to load up Python with yet another protocol, when my use case could be met by stuff that's already implemented in CPython. IOW, my motivation for saying, "hey, can't I just use this nice hook here" was to avoid asking for a *new* feature, if there weren't enough other people interested in a decoration protocol of this sort. That is, I was trying to NOT make anybody do a bunch of work on my behalf. (Clearly, I wasn't successful in the attempt, but I should at least get credit for trying. ;-)

...

Now, one minor annoyance with current class decorators is that they're *not* inherited. This is sometimes what you want, but sometimes you would prefer to automatically decorate all subclasses as well. Currently, that means writing a custom metaclass to automatically apply the decorators. This has all the problems you have noted with composability.

It seems then, that a potentially clean solution may be found by adding a *dynamic* class decoration hook. As a quick sketch of such a scheme, add the following step to the class creation process (between the current class creation process, but before the execution of lexical decorators):

for mro_cls in cls.mro(): decorators = mro_cls.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls)

Would such a dynamic class decoration hook meet your use case? Such a hook has use cases (specifically involving decorator inheritance) that *don't* require the use of sys._getframes(), so is far more likely to achieve the necessary level of consensus.

Absolutely. The main challenge with it is that I would need stateful decorators, so that they do nothing when called more than once; the motivating use cases for me do not require actual decorator inheritance. But I do have *other* stuff I'm currently using metaclasses for in 2.x, that could probably be eliminated with PEP 422. (For example, the #1 metaclass I use in 2.x is one that basically lets classes have __class_new__ and __class_init__ methods: a rough equivalent to in-line metaclasses or inheritable decorators!) In general, implementing what's effectively inherited decoration is what *most* metaclasses actually get used for, so PEP 422 is a big step forward in that. Sketching something to get a feel for the PEP... def inheritable(*decos): """Wrap a class with inheritable decorators""" def decorate(cls): cls.__decorators__ = list(decos)+list(cls.__dict__.get('__decorators__',())) for deco in reversed(decos): cls = deco(cls) return cls Hm. There are a few interesting consequences of the PEP as written. In-body decorators affect the __class__ closure (and thus super()), but out-of-body decorators don't. By me this is a good thing, but it is a bit of complexity that needs mentioning. Likewise, the need for inheritable decorators to be idempotent, in case two base classes list the same decorator. (For my own use attribute/method use cases, I can just have them remove themselves from the class's __decorators__ upon execution.) You complain that metaclasses are hard to compose, and your

...

"solution" is to monkeypatch a deliberately undocumented builtin?

To be clear, what I specifically proposed (as I mentioned in an earlier thread) was simply to patch __build_class__ in order to restore the missing __metaclass__ hook. (Which, incidentally, would make ALL code using __metaclass__ cross-version compatible between 2.x and 3.x: a potentially valuable thing in and of itself!) As for metaclasses being hard to compose, PEP 422 is definitely a step in the right direction. (Automatic metaclass combining is about the only thing that would improve it any further.)

Nick Coghlan

6 Jun 6 Jun

12:11 a.m.

On Wed, Jun 6, 2012 at 1:28 AM, PJ Eby wrote:

...

IOW, my motivation for saying, "hey, can't I just use this nice hook here" was to avoid asking for a *new* feature, if there weren't enough other people interested in a decoration protocol of this sort.

That is, I was trying to NOT make anybody do a bunch of work on my behalf. (Clearly, I wasn't successful in the attempt, but I should at least get credit for trying. ;-)

OK, I can accept that. I mainly just wanted to head off the idea of overloading __build_class__ *anyway*, even if you weren't successful in getting agreement in bringing back __metaclass__ support, or a sufficiently powerful replacement mechanism. The idea of making __build_class__ itself public *has* been discussed, and the outcome of that discussion was the creation of types.new_class() as a way to programmatically define classes that respects the new __prepare__() hook.

...

In general, implementing what's effectively inherited decoration is what *most* metaclasses actually get used for, so PEP 422 is a big step forward in that.

Yeah, that's what I realised (and will try to explain better in the next version of the PEP, based on my reply to Xavier).

...

Sketching something to get a feel for the PEP...

def inheritable(*decos): """Wrap a class with inheritable decorators""" def decorate(cls): cls.__decorators__ = list(decos)+list(cls.__dict__.get('__decorators__',())) for deco in reversed(decos): cls = deco(cls) return cls

Yep, that should work.

...

Hm. There are a few interesting consequences of the PEP as written. In-body decorators affect the __class__ closure (and thus super()), but out-of-body decorators don't. By me this is a good thing, but it is a bit of complexity that needs mentioning.

It's also worth highlighting this as a limitation of the currently supported metaclass based approach. When a metaclass runs, __class__ hasn't been filled in yet, so any attempts to call methods that use it (including via zero-argument super()) will fail. That's why my _register example at [1] ended up relying on the default argument hack: using __class__ (which is what I tried first without thinking it through) actually fails, complaining that the cell hasn't been populated. Under PEP 422, the default argument hack wouldn't be necessary, you could just write it as: def _register(cls): __class__._registry.append(cls) return cls [1] https://bitbucket.org/ncoghlan/misc/src/default/pep422.py

...

Likewise, the need for inheritable decorators to be idempotent, in case two base classes list the same decorator.

Yeah, I'll add a section on "well-behaved" dynamic decorators, which need to be aware that they may run multiple times on a single class. If they're not naturally idempotent, they may need additional code to be made so.

...

(For my own use attribute/method use cases, I can just have them remove themselves from the class's __decorators__ upon execution.)

Indeed, although that's probably unique to the approach of programmatically modifying __decorators__. Normally, if you don't want a decorator to be inherited and automatically applied to subclasses you would just use an ordinary lexical decorator.

...

...
You complain that metaclasses are hard to compose, and your "solution" is to monkeypatch a deliberately undocumented builtin?

To be clear, what I specifically proposed (as I mentioned in an earlier thread) was simply to patch __build_class__ in order to restore the missing __metaclass__ hook. (Which, incidentally, would make ALL code using __metaclass__ cross-version compatible between 2.x and 3.x: a potentially valuable thing in and of itself!)

I did briefly consider proposing that, but then I had the dynamic decorators idea which I like a *lot* more (since it also simplifies other use cases that currently require a metaclass when that isn't really what you want).

...

(Automatic metaclass combining is about the only thing that would improve it any further.)

Automatic metaclass derivation only works in practice if the metaclasses are all written to support cooperative multiple inheritance though, which is a fairly large "if" (albeit, far more likely than in the general inheritance case, since the signatures of the methods involved in type construction are significantly more constrained than those involved in instantiating arbitrary objects). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan

7:09 a.m.

On Wed, Jun 6, 2012 at 1:28 AM, PJ Eby wrote:

...

To be clear, what I specifically proposed (as I mentioned in an earlier thread) was simply to patch __build_class__ in order to restore the missing __metaclass__ hook. (Which, incidentally, would make ALL code using __metaclass__ cross-version compatible between 2.x and 3.x: a potentially valuable thing in and of itself!)

As for metaclasses being hard to compose, PEP 422 is definitely a step in the right direction. (Automatic metaclass combining is about the only thing that would improve it any further.)

Just as a warning, I doubt I'll be able to persuade enough people that this is a feature worth including in the short time left before 3.3 feature freeze. It may end up being necessary to publish metaclass and explicit decorator based variants (with their known limitations), with a view to gaining support for inclusion in 3.4. Alternatively, if people can supply examples of "post-creation manipulation only" metaclasses that could be replaced with cleaner and more composable dynamic decorator based solutions, that could help make the PEP more compelling in the near term (perhaps compelling enough to make it into 3.3). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan

9:31 a.m.

On Wed, Jun 6, 2012 at 5:09 PM, Nick Coghlan wrote:

...

On Wed, Jun 6, 2012 at 1:28 AM, PJ Eby wrote:

...
To be clear, what I specifically proposed (as I mentioned in an earlier thread) was simply to patch __build_class__ in order to restore the missing __metaclass__ hook. (Which, incidentally, would make ALL code using __metaclass__ cross-version compatible between 2.x and 3.x: a potentially valuable thing in and of itself!)

As for metaclasses being hard to compose, PEP 422 is definitely a step in the right direction. (Automatic metaclass combining is about the only thing that would improve it any further.)

Just as a warning, I doubt I'll be able to persuade enough people that this is a feature worth including in the short time left before 3.3 feature freeze. It may end up being necessary to publish metaclass and explicit decorator based variants (with their known limitations), with a view to gaining support for inclusion in 3.4.

Upgrading this warning to a fact: there's no way this topic can be given the consideration it deserves in the space of the next three weeks. I'll be changing the title of 422, spend more time discussing the problem (rather than leaping to a conclusion) and retargeting the PEP at 3.4. If you do decide to play around with monkeypatching __build_class__, please make clear to any users that it's a temporary fix until something more robust and less implementation dependent can be devised for 3.4. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

PJ Eby

3:39 p.m.

On Wed, Jun 6, 2012 at 5:31 AM, Nick Coghlan wrote:

...

...
On Wed, Jun 6, 2012 at 1:28 AM, PJ Eby wrote:

...
To be clear, what I specifically proposed (as I mentioned in an earlier thread) was simply to patch __build_class__ in order to restore the missing __metaclass__ hook. (Which, incidentally, would make ALL code using __metaclass__ cross-version compatible between 2.x and 3.x: a

...
...
valuable thing in and of itself!)

As for metaclasses being hard to compose, PEP 422 is definitely a step in the right direction. (Automatic metaclass combining is about the only

On Wed, Jun 6, 2012 at 5:09 PM, Nick Coghlan wrote: potentially thing

...
...
that would improve it any further.)

Just as a warning, I doubt I'll be able to persuade enough people that this is a feature worth including in the short time left before 3.3 feature freeze. It may end up being necessary to publish metaclass and explicit decorator based variants (with their known limitations), with a view to gaining support for inclusion in 3.4.

Upgrading this warning to a fact: there's no way this topic can be given the consideration it deserves in the space of the next three weeks. I'll be changing the title of 422, spend more time discussing the problem (rather than leaping to a conclusion) and retargeting the PEP at 3.4.

If you do decide to play around with monkeypatching __build_class__, please make clear to any users that it's a temporary fix until something more robust and less implementation dependent can be devised for 3.4.

Ideally, I would actually implement it as a backport of the PEP... in which case I suppose making it part of the class creation machinery (vs. embedding it in type.__call__ or some place like that) will make that process easier. Again, as I said earlier, I'm talking about this now because there was related discussion now, not because I'm actively trying to port my libraries. At this point, I've only done a few "make this library usable from 3.x as-is" changes by user request, for some of my smaller libraries that were mostly there already (e.g. simplegeneric).

Joao S. O. Bueno

5 Jun 5 Jun

12:21 p.m.

On 4 June 2012 21:10, PJ Eby wrote:

...

...
...
I only use __metaclass__ in 2.x for this because it's the only way for code executed in a class body to gain access to the class at creation time.

PJ, it maybe just me, but what does this code do that can't be done at the metaclass' __new__ method? You might have to rewrite some method-decorators, so that they just mark a method at class body execution time, and then, whatever the decorator used to do at this time, would be done at meta's __new__ - I have this working in some code (and in Python 2 already). js -><-

4335

Age (days ago)

4337

Last active (days ago)

List overview

Download

22 comments

7 participants

participants (7)

Eric Snow
Ethan Furman
Joao S. O. Bueno
Michael Foord
Nick Coghlan
PJ Eby
Xavier Morel

Possible rough edges in Python 3 metaclasses (was Re: Language reference updated for metaclasses)

tags

participants (7)