PEP 487 vs 422 (dynamic class decoration)
I recently got an inquiry from some of my users about porting some of my libraries to Python 3 that make use of the Python 2 __metaclass__ facility. While checking up on the status of PEP 422 today, I found out about its recently proposed replacement, PEP 487. While PEP 487 is a generally fine PEP, it actually *rules out* the specific use case that I wanted PEP 422 for in the first place: dynamic addition of callbacks or decorators for use at class creation time without requiring explicit inheritance or metaclass participation. (So that e.g. method decorators can access the enclosing class at class definition time.) As discussed previously prior to the creation of PEP 422, it is not possible to port certain features of my libraries to work on Python 3 without some form of that ability, and the only thing that I know of that could even *potentially* provide that ability outside of PEP 422 is monkeypatching __build_class__ (which might not even work). That is, the very thing that PEP 422 was created to avoid the need for. ;-) One possible alteration would be to replace __init_subclass__ with some sort of __init_class__ invoked on the class that provides it, not just subclasses. That would allow the kind of dynamic decoration that PEP 422 allows. However, this approach was rather specifically ruled out in earlier consideration of PEP 422, so.... Another alternative would be to have the default __init_subclass__ look at a class-level __decorators__ attribute, as originally discussed for PEP 422. That would solve *my* problem, but feels too much like adding more than One Way To Do It. So... honestly, I'm not sure where to go from here. Is there any chance that this is going to be changed, or revert to the PEP 422 approach, or... something? If so, what Python version will the "something" be in? Or is this use case just going to be a dead parrot in Python 3, period?
On 2 April 2015 at 07:35, PJ Eby <pje@telecommunity.com> wrote:
How hard is the requirement against relying on a mixin class or class decorator to request the defining class aware method decorator support? Is the main concern with the fact that failing to apply the right decorator/mixin at the class level becomes a potentially silent failure where the class aware method decorators aren't invoked properly?
My preference at this point would definitely be to introduce a mixin class into the affected libraries and frameworks with an appropriate PEP 487 style __init_subclass__ that was a noop in Python 2 (which would rely on metaclass injection instead), but implemented the necessary "defining class aware" method decorator support in Python 3. The question of dynamically injecting additional base classes from the class body to allow the use of certain method decorators to imply specific class level behaviour could then be addressed as a separate proposal (e.g. making the case for an "__append_mixins__" attribute), rather than being linked directly to the question of how we going about defining inherited creation time behaviour without needing a custom metaclass. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Apr 1, 2015 at 10:39 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The concern is twofold: it breaks proper information hiding/DRY, *and* it fails silently. It should not be necessary for clients of package A1 (that uses a decorator built using package B2) to mixin a metaclass or decorator from package C3 (because B2 implemented its decorators using C3), just for package A1's decorator to work properly in the *client package's class*. (And then, of course, this all silently breaks if you forget, and the breakage might happen at the A1, B2, or C3 level.) Without a way to hook into the class creation process, there is no way to verify correctness and prevent the error from passing silently. (OTOH, if there *is* a way to hook into the creation process, the problem is solved: there's no need to mix anything in anyway, because the hook can do whatever the mixin was supposed to do.) The only way PEP 487 could be a solution is if the default `object.__init_subclass__` supported one of the earlier __decorators__ or __autodecorate__ proposals, or if the PEP were for an `__init_class__` that operated on the defining class, instead of operating only on subclasses. (I need to hook the creation of a class that's *being defined*, not the definition of its future subclasses.)
If this were suitable for the use case, I'd have done it already. DecoratorTools has had a mixin that provides a __class_init__ feature since 2007, which could be ported to Python 3 in a straighforward manner as a third-party module. (It's just a mixin that provides a metaclass; under 3.x it could probably just be a plain metaclass with no mixin.)
Then maybe we should do that first, since PEP 487 doesn't do anything you can't *already* do with a mixin, all the way back to Python 2.2. IOW, there's no need to modify the core just to have *that* feature, since if you control the base class you can already do what PEP 487 does in essentially every version of Python, ever. If that's all PEP 487 is going to do, it should just be a PyPI package on a stdlib-inclusion track, not a change to core Python. It's not actually adding back any of the dynamicness (dynamicity? hookability?) that PEP 3115 took away.
On 2 April 2015 at 16:38, PJ Eby <pje@telecommunity.com> wrote:
The specific feature that PEP 487 is adding is the ability to customise creation of subclasses without risking the introduction of a metaclass conflict. That allows it to be used in situations where adopting any of the existing metaclass based mechanisms would require a potential compatibility break (as well as being far more approachable as a mechanism than the use of custom metaclasses). The gap I agree this approach leaves is a final post-namespace-execution step that supports establishing any class level invariants implied by decorators and other functions used in the class body. Python 2 allowed that to be handled with a dynamically generated __metaclass__ and PEP 422 through __autodecorate__, while PEP 487 currently has no equivalent mechanism. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Apr 02, 2015 at 06:46:11PM +1000, Nick Coghlan wrote:
On 2 April 2015 at 16:38, PJ Eby <pje@telecommunity.com> wrote:
Perhaps PEP 422 should be back in the running then (possible reduced in scope, I haven't read it in a while), along with PEP 487, since they seem to target different areas. -- ~Ethan~
On Thu, Apr 2, 2015 at 4:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
But metaclass conflicts are *also* fixable in end-user code, and have been since 2.2. All you need to do is use a metaclass *function* that automatically merges the metaclasses involved, which essentially amounts to doing `class MergedMeta(base1.__class__, base2.__class__,...)`. (Indeed, I've had a library for doing just that since 2002, that originally ran on Python 2.2,.) On Python 3, it's even easier to use that approach, because you can just use something like `class whatever(base1, base2, metaclass=noconflict)` whenever a conflict comes up. (And because the implementation wouldn't have to deal with classic classes or __metaclass__, as my Python 2 implementation has to.) IOW, *all* of PEP 487 is straightforward to implement in userspace as a metaclass and a function that already exist off-the-shelf in Python 2... and whose implementations would be simplified by porting them to Python 3, and dropping any extraneous features: * http://svn.eby-sarna.com/PEAK/src/peak/util/Meta.py?view=markup (the `makeClass` function does what my hypothetical `noconflict` above does, with a slightly different API, and support for classic classes, __metaclass__, etc., that could all be stripped out) * http://svn.eby-sarna.com/DecoratorTools/peak/util/decorators.py?view=markup (see the `classy_class` metaclass and `classy` mixin base that implement features similar to `__init_subclass__`, plus others that could be stripped out) Basically, you can pull out those functions/classes (and whatever else they use in those modules), port 'em to Python 3, make any API changes deemed suitable, and call it a day. And the resulting code could go to a stdlib metaclass utility module after a reasonable break-in period.
(as well as being far more approachable as a mechanism than the use of custom metaclasses).
Sure, nobody's arguing that it's not a desirable feature. I *implemented* that mechanism for Python 2 (eight years ago) because it's easier to use even for those of us who are fully versed in the dark metaclass arts. ;-) Here's the documentation: http://peak.telecommunity.com/DevCenter/DecoratorTools#meta-less-classes So the feature doesn't even require *stdlib* adoption, let alone changes to Python core. (Heck, I wasn't even the first to implement this feature: Zope had it for Python *1.5.2*, in their ExtensionClass.) It's a totally solved problem in Python 2, although the solution is admittedly not widely known. If the PEP 487 metaclass library, however, were to just port some bits of my code to Python 3 this could be a done deal already and available in *all* versions of Python 3, not just the next one.
Right. And it's *only* having such a mechanism available by *default* that requires a language change. Conversely, if we *are* making a language change, then adding a hook that allows method decorators to access the just-defined class provides roughly the same generality that Python 2 had in this respect. All I want is the ability for method decorators to find out what class they were added to, at the time the class is built, rather than having to wait for an access or invocation that may never come. This could be as simple as __build_class__ or type.__call__ looking through the new class's dictionary for objects with a `__used_in_class__(cls, name)` method, e.g.: for k, v in dict.items(): if hasattr(v, '__used_in_class__'): v.__used_in_class__(cls, k) This doesn't do what PEP 487 or 422 do, but it's the bare minimum for what I need, and it actually allows this type of decorator to avoid any frame inspection because they can just add a __used_in_class__ attribute to the functions being decorated. (It actually also eliminates another common use for metaclasses and class decorators, e.g. in ORM and other structure-like classes that need properties to know their names without duplicating code, as in `x = field(int)` vs `x = field('x', int)`. I have a fair amount of code that does this sort of thing, too, though it's not in urgent need of porting to Python 3.)
Hi everyone, for those new to the discussion, I am the author of PEP 487, which has been discussed here: https://mail.python.org/pipermail/python-ideas/2015-February/032249.html
I am just not capable to understand things at such an abstract level, would it be possible to give a simple example on what and how you did things in python 2?
That was my point. You can find the PyPI package at: https://pypi.python.org/pypi/metaclass Greetings Martin
On 2 April 2015 at 07:35, PJ Eby <pje@telecommunity.com> wrote:
How hard is the requirement against relying on a mixin class or class decorator to request the defining class aware method decorator support? Is the main concern with the fact that failing to apply the right decorator/mixin at the class level becomes a potentially silent failure where the class aware method decorators aren't invoked properly?
My preference at this point would definitely be to introduce a mixin class into the affected libraries and frameworks with an appropriate PEP 487 style __init_subclass__ that was a noop in Python 2 (which would rely on metaclass injection instead), but implemented the necessary "defining class aware" method decorator support in Python 3. The question of dynamically injecting additional base classes from the class body to allow the use of certain method decorators to imply specific class level behaviour could then be addressed as a separate proposal (e.g. making the case for an "__append_mixins__" attribute), rather than being linked directly to the question of how we going about defining inherited creation time behaviour without needing a custom metaclass. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Apr 1, 2015 at 10:39 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The concern is twofold: it breaks proper information hiding/DRY, *and* it fails silently. It should not be necessary for clients of package A1 (that uses a decorator built using package B2) to mixin a metaclass or decorator from package C3 (because B2 implemented its decorators using C3), just for package A1's decorator to work properly in the *client package's class*. (And then, of course, this all silently breaks if you forget, and the breakage might happen at the A1, B2, or C3 level.) Without a way to hook into the class creation process, there is no way to verify correctness and prevent the error from passing silently. (OTOH, if there *is* a way to hook into the creation process, the problem is solved: there's no need to mix anything in anyway, because the hook can do whatever the mixin was supposed to do.) The only way PEP 487 could be a solution is if the default `object.__init_subclass__` supported one of the earlier __decorators__ or __autodecorate__ proposals, or if the PEP were for an `__init_class__` that operated on the defining class, instead of operating only on subclasses. (I need to hook the creation of a class that's *being defined*, not the definition of its future subclasses.)
If this were suitable for the use case, I'd have done it already. DecoratorTools has had a mixin that provides a __class_init__ feature since 2007, which could be ported to Python 3 in a straighforward manner as a third-party module. (It's just a mixin that provides a metaclass; under 3.x it could probably just be a plain metaclass with no mixin.)
Then maybe we should do that first, since PEP 487 doesn't do anything you can't *already* do with a mixin, all the way back to Python 2.2. IOW, there's no need to modify the core just to have *that* feature, since if you control the base class you can already do what PEP 487 does in essentially every version of Python, ever. If that's all PEP 487 is going to do, it should just be a PyPI package on a stdlib-inclusion track, not a change to core Python. It's not actually adding back any of the dynamicness (dynamicity? hookability?) that PEP 3115 took away.
On 2 April 2015 at 16:38, PJ Eby <pje@telecommunity.com> wrote:
The specific feature that PEP 487 is adding is the ability to customise creation of subclasses without risking the introduction of a metaclass conflict. That allows it to be used in situations where adopting any of the existing metaclass based mechanisms would require a potential compatibility break (as well as being far more approachable as a mechanism than the use of custom metaclasses). The gap I agree this approach leaves is a final post-namespace-execution step that supports establishing any class level invariants implied by decorators and other functions used in the class body. Python 2 allowed that to be handled with a dynamically generated __metaclass__ and PEP 422 through __autodecorate__, while PEP 487 currently has no equivalent mechanism. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Apr 02, 2015 at 06:46:11PM +1000, Nick Coghlan wrote:
On 2 April 2015 at 16:38, PJ Eby <pje@telecommunity.com> wrote:
Perhaps PEP 422 should be back in the running then (possible reduced in scope, I haven't read it in a while), along with PEP 487, since they seem to target different areas. -- ~Ethan~
On Thu, Apr 2, 2015 at 4:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
But metaclass conflicts are *also* fixable in end-user code, and have been since 2.2. All you need to do is use a metaclass *function* that automatically merges the metaclasses involved, which essentially amounts to doing `class MergedMeta(base1.__class__, base2.__class__,...)`. (Indeed, I've had a library for doing just that since 2002, that originally ran on Python 2.2,.) On Python 3, it's even easier to use that approach, because you can just use something like `class whatever(base1, base2, metaclass=noconflict)` whenever a conflict comes up. (And because the implementation wouldn't have to deal with classic classes or __metaclass__, as my Python 2 implementation has to.) IOW, *all* of PEP 487 is straightforward to implement in userspace as a metaclass and a function that already exist off-the-shelf in Python 2... and whose implementations would be simplified by porting them to Python 3, and dropping any extraneous features: * http://svn.eby-sarna.com/PEAK/src/peak/util/Meta.py?view=markup (the `makeClass` function does what my hypothetical `noconflict` above does, with a slightly different API, and support for classic classes, __metaclass__, etc., that could all be stripped out) * http://svn.eby-sarna.com/DecoratorTools/peak/util/decorators.py?view=markup (see the `classy_class` metaclass and `classy` mixin base that implement features similar to `__init_subclass__`, plus others that could be stripped out) Basically, you can pull out those functions/classes (and whatever else they use in those modules), port 'em to Python 3, make any API changes deemed suitable, and call it a day. And the resulting code could go to a stdlib metaclass utility module after a reasonable break-in period.
(as well as being far more approachable as a mechanism than the use of custom metaclasses).
Sure, nobody's arguing that it's not a desirable feature. I *implemented* that mechanism for Python 2 (eight years ago) because it's easier to use even for those of us who are fully versed in the dark metaclass arts. ;-) Here's the documentation: http://peak.telecommunity.com/DevCenter/DecoratorTools#meta-less-classes So the feature doesn't even require *stdlib* adoption, let alone changes to Python core. (Heck, I wasn't even the first to implement this feature: Zope had it for Python *1.5.2*, in their ExtensionClass.) It's a totally solved problem in Python 2, although the solution is admittedly not widely known. If the PEP 487 metaclass library, however, were to just port some bits of my code to Python 3 this could be a done deal already and available in *all* versions of Python 3, not just the next one.
Right. And it's *only* having such a mechanism available by *default* that requires a language change. Conversely, if we *are* making a language change, then adding a hook that allows method decorators to access the just-defined class provides roughly the same generality that Python 2 had in this respect. All I want is the ability for method decorators to find out what class they were added to, at the time the class is built, rather than having to wait for an access or invocation that may never come. This could be as simple as __build_class__ or type.__call__ looking through the new class's dictionary for objects with a `__used_in_class__(cls, name)` method, e.g.: for k, v in dict.items(): if hasattr(v, '__used_in_class__'): v.__used_in_class__(cls, k) This doesn't do what PEP 487 or 422 do, but it's the bare minimum for what I need, and it actually allows this type of decorator to avoid any frame inspection because they can just add a __used_in_class__ attribute to the functions being decorated. (It actually also eliminates another common use for metaclasses and class decorators, e.g. in ORM and other structure-like classes that need properties to know their names without duplicating code, as in `x = field(int)` vs `x = field('x', int)`. I have a fair amount of code that does this sort of thing, too, though it's not in urgent need of porting to Python 3.)
Hi everyone, for those new to the discussion, I am the author of PEP 487, which has been discussed here: https://mail.python.org/pipermail/python-ideas/2015-February/032249.html
I am just not capable to understand things at such an abstract level, would it be possible to give a simple example on what and how you did things in python 2?
That was my point. You can find the PyPI package at: https://pypi.python.org/pypi/metaclass Greetings Martin
participants (4)
-
Ethan Furman -
Martin Teichmann -
Nick Coghlan -
PJ Eby