[Python-Dev] Possible rough edges in Python 3 metaclasses (was Re: Language reference updated for metaclasses)

[PJ Eby]

On Mon, Jun 4, 2012 at 7:18 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Tue, Jun 5, 2012 at 8:58 AM, PJ Eby <pje at telecommunity.com> wrote:
> > On Mon, Jun 4, 2012 at 6:15 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >>
> >> It's actually the pre-decoration class, since the cell is initialised
> >> before the class is passed to the first decorator. I agree it's a little
> >> weird, but I did try to describe it accurately in the new docs.
> >
> > I see that now; it might be helpful to explicitly call that out.
> >
> > This is adding to my list of Python 3 metaclass gripes, though.  In
> Python
> > 2, I have in-the-body-of-a-class decorators implemented using
> metaclasses,
> > that will no longer work because of PEP 3115...
>
> I'm not quite following this one - do you mean they won't support
> __prepare__, won't play nicely with other metaclasses that implement
> __prepare__, or something else?
>

I mean that class-level __metaclass__ is no longer supported as of PEP
3115, so I can't use that as a way to non-invasively obtain the enclosing
class at class creation time.

(Unfortunately, I didn't realize until relatively recently that it wasn't
supported any more; the PEP itself doesn't say the functionality will be
removed.  Otherwise, I'd have lobbied sooner for a better migration path.)



> > Meanwhile, mixing metaclasses is more difficult than ever, due to
> > __prepare__, and none of these flaws can be worked around officially,
> > because __build_class__ is an "implementation detail".  I *really* want
> to
> > like Python 3, but am still hoping against hope for the restoration of
> hooks
> > that __metaclass__ allowed, or some alternative mechanism that would
> serve
> > the same use cases.
> >
> > Specifically, my main use case is method-level decorators and attribute
> > descriptors that need to interact with a user-defined class, *without*
> > requiring that user-defined class to either 1) redundantly decorate the
> > class or 2) inherit from some specific base or inject a specific
> metaclass.
> > I only use __metaclass__ in 2.x for this because it's the only way for
> code
> > executed in a class body to gain access to the class at creation time.
> >
> > The reason for wanting this to be transparent is that 1) if you forget
> the
> > redundant class-decorator, mixin, or metaclass, stuff will silently not
> > work,
>
> Why would it silently not work? What's preventing you from having
> decorators that create wrapped functions that fail noisily when
> called, then providing a class decorator that unwraps those functions,
> fixes them up with the class references they need and stores the
> unwrapped and updated versions back on the class.
>

If you are registering functions or attributes in some type of registry
either at the class level or in some sort of global registry, then the
function will never be called or the attribute accessed, so there is no
opportunity to give an error message saying, "you should have done this".
Something will simply fail to happen that should have happened.

Essentially, the lack of this hook forces things to be done outside of
class bodies that make more sense to be done *inside* class bodies.


> and 2) mixing bases or metaclasses has much higher coupling to the
> > library providing the decorators or descriptors, and greatly increases
> the
> > likelihood of mixing metaclasses.
>
> So don't do that, then. Be explicit.
>

Easy for you to say.  However, I have *many* decorators and descriptors
which follow this pattern, in various separately-distributed libraries.  If
*each* of these libraries now grows a redundant class decorator for Python
3, then any code which uses more than one in the same class will now have a
big stack of decorator noise on top of it...  and missing any one of them
will cause silent failure.

And of course, all other libraries which use these decorators and
descriptors will *also* have to add this line-noise in order to port to
Python 3...  including the applications that depend on those libraries.
And let's not forget the people who subclass or extend any of my decorators
or descriptors -- they will need to tell *their* users to begin adding
redundant class-level decorators, and so on.



> > And at the moment, the only workaround I can come up with that *doesn't*
> > involve replacing __build_class__ is abusing the system trace hook; ISTM
> > that replacing __build_class__ is the better of those two options.
>
> Stop trying to be implicit. Implicit magic sucks,


The fact that a method-level decorator or descriptor might need access to
its containing class when the class is created does not in any way make it
*intrinsically* "magical" or "implicit".  The focus here is on the
decorator or descriptor: the class access is just an implementation detail,
and generally doesn't involve modifying the class itself.

So, please let's not start FUDding here.  None of the descriptors or
decorators I'm describing are implicit or magical in the least.  They are
documented as doing what they do, and they use perfectly legal mechanisms
of Python 2 to implement those behaviors.  If tomorrow a new PEP were
introduced that provided an alternate way of doing this, I'd gladly use it
to migrate these features forward.

Implying that my use cases "suck" does not help; I could just as easily
say, "redundancy sucks", and we are no further along to a solution.  It
might be more helpful to propose some alternate mechanism to provide the
same feature to be introduced in a future Python version, like a special
__decorators__ key in a class dictionary that the default type would
iterate over and call with the class.  Or perhaps a proposal that the
default type creation code simply iterate over the contents of the class
namespace and call "ob.__used_in_class__(cls, attrname)" on each found
object.  Or something else altogether, any of which I would happily use in
place of the old __metaclass__ hook, and which would certainly be less
obscure.


> At this point, with the additions of types.new_class(), ISTM that every
> > Python implementation will have to *have* a __build_class__ function or
> its
> > equivalent; all that remains is the question of whether they allow
> > *replacing* it.
>
> types.new_class() is actually a pure Python reimplementation of the
> PEP 3115 algorithm. Why would it imply the need for a __build_class__
> function?
>

Because now every Python implementation will contain the code to do this
*twice*: once inside its normal class creation machinery, and once inside
of types.new_class().  (Well, unless they share code internally, of course.)

IOW, saying that __build_class__ is an implementation detail of CPython
doesn't quite wash: thanks to types.new_class() every Python implementation
now *must* support programmatically creating a class in this way.  The only
detail that remains is whether it's allowed to replace the built-in
equivalent to __build_class__, and whether it's documented as a standard
feature ala __import__ (and its __metaclass__ predecessor).

Note, though, that I've only focused on __build_class__ because it's
low-hanging fruit: it could be used to work around the absence of
__metaclass__ or a more-specific hook like the ones I mentioned above,
*and* it's already implemented in Python 3.x.  (That doesn't make it the
best solution, of course, just a low-hanging one.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120604/ac4931de/attachment.html>