[Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[Nick Coghlan]

On 3 April 2015 at 11:32, PJ Eby <pje at telecommunity.com> wrote:
> On Thu, Apr 2, 2015 at 6:24 PM, Martin Teichmann
> <lkb.teichmann at gmail.com> wrote:
>> The whole point of PEP 487 was to reduce PEP 422 so much that
>> it can be written in python and back-ported.
>
> As I said earlier, it's a fine feature and should be in the stdlib for
> Python 3.  (But it should have a `noconflict` feature added, and it
> doesn't need a language change.)
>
> However, since my specific use case was the one PEP 422 was originally
> written to solve, and PEP 487 does not address that use case, it is
> not a suitable substitute *for PEP 422*.
>
> This is also not your fault; you didn't force Nick to withdraw it,
> after all.  ;-)
>
> My main concern in this thread, however, is ensuring that either the
> use case behind PEP 422 doesn't get dropped, or that Nick is now okay
> with me implementing that feature by monkeypatching __build_class__.
> Since he practically begged me not to do that in 2012, and IIRC
> *specifically created* PEP 422 to provide an alternative way for me to
> accomplish this *specific* use case, I wanted to see what his current
> take was.  (That is, did he forget the history of the PEP, or does he
> no longer care about userspace code hooking __build_class__?  Is there
> some other proposal that would be a viable alternative? etc.)

It's actually both - I'd forgotten the origins of PEP 422, *and* I've
come around to the view that this level of implicit behaviour is a bad
idea because it's inherently confusing. The various incarnations of
PEP 422 ended up remaining at least somewhat confusing for *me* and I
wrote it. The "what happens when?" story in PEP 487 is much simpler.

That means I'm now OK with monkeypatching __build_class__ being the
only way to get dynamic hooking of the class currently being defined
from the class body - folks that really want that behaviour can
monkeypatch it in, while folks that think it's a bad idea don't need
to worry about.

>> Now you want to be able to write decorators whose details
>> are filled in at class creation time.
>
> Not "now"; it's been possible to do this in Python 2 for over a
> decade, and code that does so is in current use by other packages.
> The package providing this feature (DecoratorTools) was downloaded 145
> times today, and 3274 times in the past month, so there is active,
> current use of it by other Python 2 packages.  (Though I don't know
> how many of them depend directly or indirectly upon this particular
> feature.)
>
> Currently, however, it is not possible to port this feature of
> DecoratorTools (or any other package that uses that feature,
> recursively) to Python 3, due to the removal of __metaclass__ and the
> lack of any suitable substitute hook.

Right, any post-namespace-execution modification of class definitions
in Python 3 currently has to be declared in the class definition
header, either as an explicit decorator, as a metaclass declaration,
or through inheritance from a particular base class.

PEP 487 doesn't change that, it just adds a way for the last case to
work without needing a custom metaclass in the picture.

PEP 422 *did* change that, by providing a way for the namespace itself
to register a post-execution hook, akin to one of the way's
__metaclass__ could be used in Python 2.

What's changed since I first wrote PEP 422 is that I've come to view
the requirement to be explicit in Python 3 as a gain for readability,
rather than as a limitation to be eliminated - there will always be
*some* hint in the class header that post-namespace-execution
modifications may be happening, even if it's just having a declared
base class other than object.

>> Your point is that you want to be able to use your decorators
>> without having to ask users to also inherit a specific class.
>> I personally don't think that's desirable.  Many frameworks out
>> there have such kind of decorators and mandatory base classes
>> and that works fine.
>
> The intended use case is for generic method decorators that have
> nothing to do with the base class per se, so inheriting from a
> specific base-class is an anti-feature in this case.

If that's the use case, it would be preferable to explore enhancing
the type based dispatch capabilities in functools as a follow-up to
Guido's work on type hinting enhacements.

>> The only problem remains once you need to
>> inherit more than one of those classes, as their metaclasses
>> most likely clash. This is what PEP 487 fixes.
>
> No, it addresses the issue for certain *specific* metaclass use cases.
> It does not solve the problem of metaclass conflict in general; for
> that you need something like the sample `noconflict` code I posted,
> which works for Python 3.1+ and doesn't require a language change.

Neither PEP 422 nor 487 are designed to eliminate metaclass conflicts
in general, they're primarily designed to let base classes run
arbitrary code after the namespace has been executed in a subclass
definition *without* needing a custom metaclass. They both introduce a
new tier in the metaprogramming hierarchy between explicit class
decorators and full custom metaclasses.

>> So my opinion is that it is not too hard a requirement to ask
>> a user to inherit a specific mixin class for the sake of using
>> a decorator.
>
> If this logic were applied to PEP 487 as it currently stands, the PEP
> should be rejected, since its use case is even *more* easily
> accomplished by inheriting from a specific mixin class.  (Since the
> feature only works on subclasses anyway!)

No, you can't do it currently without risking a backwards
incompatibility through the introduction of a custom metaclass.

> Further, if the claim is that metaclass conflict potential makes PEP
> 487 worthy of a language change, then by the same logic method
> decorators are just as worthy of a language change, since any mixin
> required to use a method decorator would be *just as susceptible* to
> metaclass conflicts as SubclassInit.

There wouldn't be a custom metaclass involved in the native
implementation of PEP 487, only in the backport.

> Finally, I of course disagree with the conclusion that it's okay to
> require mixins in order for method decorators to access the containing
> class, since it is not a requirement in Python 2, due to the
> availability of the __metaclass__ hook.

Right, that's the crux of the disagreement: is the fact that you can
inject a postprocessor into a Python 2 class namespace while it is
being executed a good thing, or does it inherently lead to surprising
code where it isn't clear where the postprocessing injection happened?

When I first wrote PEP 422 I was of the view that "Python 2 allows
class definition postprocessing injection, we should allow it in
Python 3 as well". I've since changed my view to "Having to declare
post-processing of a class definition up front as a decorator, base
class or metaclass is a good thing for readability, as otherwise
there's nothing obvious when reading a class definition that tells you
whether or not postprocessing may happen, so you have to assume its
possible for *every* class definition".

> Further, PEP 422 was
> previously approved to fix this problem, and has a patch in progress,
> so I'm understandably upset by its sudden withdrawal and lack of
> suitable replacement.

PEP 422 has never been approved - it's only ever been Draft or
Deferred (and now Withdrawn). If you would like to take it over and
champion it as a competitor to PEP 487, I'd be fine with that, I just
no longer think it's a good idea myself.

> At this point, though, I mostly just want to get some kind of closure.
> After three years, I'd like to know if this is a yea or nay, so I can
> port the thing and move on, whether it's through a standardized
> mechanism or ugly monkeypatching.  Honestly, the only reason I'm even
> discussing this in the first place is because 1) Nick pleaded with me
> three years ago not to hold off porting until a standardized way of
> doing this could be added to the language,

My apologies for leading you up the garden path - back then I
genuinely thought bringing back the capability was a good idea, but
the combination of the original deferral to process the previous lot
of feedback, and the more recent discussions with Martin that led to
the creation of PEP 487 and the withdrawal of PEP 422 changed my mind.

> and 2) I've had some recent
> inquiries from users about porting PEAK-Rules (which uses this
> particular feature of DecoratorTools) to Python 3.  So I went to check
> on PEP 422's status, and here we are.

Given my change of heart, I believe that at this point, if you were
willing to champion a revived PEP 422 that implemented the behaviour
you're after, that would be ideal, with monkeypatching the desired
behaviour in as a fallback plan if the PEP is still ultimately
rejected. Alternatively, you could go the monkeypatching path first,
and then potentially seek standardisation later after you've had some
practical experience with it - I now consider it an orthogonal
capability to the feature in PEP 487, so the acceptance of the latter
wouldn't necessarily preclude acceptance of a hook for class
postprocessing injection.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia