Providing a mechanism for PEP 3115 compliant dynamic class creation
In reviewing a fix for the metaclass calculation in __build_class__ [1], I realised that PEP 3115 poses a potential problem for the common practice of using "type(name, bases, ns)" for dynamic class creation. Specifically, if one of the base classes has a metaclass with a significant __prepare__() method, then the current idiom will do the wrong thing (and most likely fail as a result), since "ns" will probably be an ordinary dictionary instead of whatever __prepare__() would have returned. Initially I was going to suggest making __build_class__ part of the language definition rather than a CPython implementation detail, but then I realised that various CPython specific elements in its signature made that a bad idea. Instead, I'm thinking along the lines of an "operator.prepare(metaclass, bases)" function that does the metaclass calculation dance, invoking __prepare__() and returning the result if it exists, otherwise returning an ordinary dict. Under the hood we would refactor this so that operator.prepare and __build_class__ were using a shared implementation of the functionality at the C level - it may even be advisable to expose that implementation via the C API as PyType_PrepareNamespace(). The correct idiom for dynamic type creation in a PEP 3115 world would then be: from operator import prepare cls = type(name, bases, prepare(type, bases)) Thoughts? Cheers, Nick. [1] http://bugs.python.org/issue1294232 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Apr 19, 2011 at 16:10, Nick Coghlan <ncoghlan@gmail.com> wrote:
In reviewing a fix for the metaclass calculation in __build_class__ [1], I realised that PEP 3115 poses a potential problem for the common practice of using "type(name, bases, ns)" for dynamic class creation.
Specifically, if one of the base classes has a metaclass with a significant __prepare__() method, then the current idiom will do the wrong thing (and most likely fail as a result), since "ns" will probably be an ordinary dictionary instead of whatever __prepare__() would have returned.
Initially I was going to suggest making __build_class__ part of the language definition rather than a CPython implementation detail, but then I realised that various CPython specific elements in its signature made that a bad idea.
Are you referring to the first 'func' argument? (Which is basically the body of the "class" statement, if I'm not mistaken).
Instead, I'm thinking along the lines of an "operator.prepare(metaclass, bases)" function that does the metaclass calculation dance, invoking __prepare__() and returning the result if it exists, otherwise returning an ordinary dict. Under the hood we would refactor this so that operator.prepare and __build_class__ were using a shared implementation of the functionality at the C level - it may even be advisable to expose that implementation via the C API as PyType_PrepareNamespace().
__prepare__ also needs the name and optional keyword arguments. So it probably should be something like "operator.prepare(name, bases, metaclass, **kw)". But this way it would need almost the same arguments as __build_class__(func, name, *bases, metaclass=None, **kwds).
The correct idiom for dynamic type creation in a PEP 3115 world would then be:
from operator import prepare cls = type(name, bases, prepare(type, bases))
Thoughts?
When creating a dynamic type, we may want to do it with a non-empty namespace. Maybe like this (with the extra arguments mentioned above): from operator import prepare ns = prepare(name, bases, type, **kwargs) ns.update(my_ns) # add the attributes we want cls = type(name, bases, ns) What about an "operator.build_class(name, bases, ns, **kw)" function? It would work like this: def build_class(name, bases, ns, **kw): metaclass = kw.pop('metaclass', type) pns = prepare(name, bases, metaclass, **kw) pns.update(ns) return metaclass(name, bases, pns) (Where 'prepare' is the same as above). This way we wouldn't even need to make 'prepare' public, and the new way to create a dynamic type would be: from operator import build_class cls = build_class(name, bases, ns, **my_kwargs) Daniel
/me pages thoughts from 12 months ago back into brain... On Sun, Apr 15, 2012 at 7:36 PM, Daniel Urban <urban.dani+py@gmail.com> wrote:
On Tue, Apr 19, 2011 at 16:10, Nick Coghlan <ncoghlan@gmail.com> wrote:
Initially I was going to suggest making __build_class__ part of the language definition rather than a CPython implementation detail, but then I realised that various CPython specific elements in its signature made that a bad idea.
Are you referring to the first 'func' argument? (Which is basically the body of the "class" statement, if I'm not mistaken).
Yup, I believe that was my main objection to exposing __build_class__ directly. There's no obligation for implementations to build a throwaway function to evaluate a class body.
__prepare__ also needs the name and optional keyword arguments. So it probably should be something like "operator.prepare(name, bases, metaclass, **kw)". But this way it would need almost the same arguments as __build_class__(func, name, *bases, metaclass=None, **kwds).
True.
The correct idiom for dynamic type creation in a PEP 3115 world would then be:
from operator import prepare cls = type(name, bases, prepare(type, bases))
Thoughts?
When creating a dynamic type, we may want to do it with a non-empty namespace. Maybe like this (with the extra arguments mentioned above):
from operator import prepare ns = prepare(name, bases, type, **kwargs) ns.update(my_ns) # add the attributes we want cls = type(name, bases, ns)
What about an "operator.build_class(name, bases, ns, **kw)" function? It would work like this:
def build_class(name, bases, ns, **kw): metaclass = kw.pop('metaclass', type) pns = prepare(name, bases, metaclass, **kw) pns.update(ns) return metaclass(name, bases, pns)
(Where 'prepare' is the same as above). This way we wouldn't even need to make 'prepare' public, and the new way to create a dynamic type would be:
from operator import build_class cls = build_class(name, bases, ns, **my_kwargs)
No, I think we would want to expose the created namespace directly - that way people can use update(), direct assigment, exec(), eval(), or whatever other mechanism they choose to handle the task of populating the namespace. However, a potentially cleaner way to do that might be offer use an optional callback API rather than exposing a separate public prepare() function. Something like: def build_class(name, bases=(), kwds=None, eval_body=None): metaclass, ns = _prepare(name, bases, kwds) if eval_body is not None: eval_body(ns) return metaclass(name, bases, ns) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Apr 15, 2012 at 13:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
/me pages thoughts from 12 months ago back into brain...
Sorry about that, I planned to do this earlier...
On Sun, Apr 15, 2012 at 7:36 PM, Daniel Urban <urban.dani+py@gmail.com> wrote:
On Tue, Apr 19, 2011 at 16:10, Nick Coghlan <ncoghlan@gmail.com> wrote:
Initially I was going to suggest making __build_class__ part of the language definition rather than a CPython implementation detail, but then I realised that various CPython specific elements in its signature made that a bad idea.
Are you referring to the first 'func' argument? (Which is basically the body of the "class" statement, if I'm not mistaken).
Yup, I believe that was my main objection to exposing __build_class__ directly. There's no obligation for implementations to build a throwaway function to evaluate a class body.
__prepare__ also needs the name and optional keyword arguments. So it probably should be something like "operator.prepare(name, bases, metaclass, **kw)". But this way it would need almost the same arguments as __build_class__(func, name, *bases, metaclass=None, **kwds).
True.
The correct idiom for dynamic type creation in a PEP 3115 world would then be:
from operator import prepare cls = type(name, bases, prepare(type, bases))
Thoughts?
When creating a dynamic type, we may want to do it with a non-empty namespace. Maybe like this (with the extra arguments mentioned above):
from operator import prepare ns = prepare(name, bases, type, **kwargs) ns.update(my_ns) # add the attributes we want cls = type(name, bases, ns)
What about an "operator.build_class(name, bases, ns, **kw)" function? It would work like this:
def build_class(name, bases, ns, **kw): metaclass = kw.pop('metaclass', type) pns = prepare(name, bases, metaclass, **kw) pns.update(ns) return metaclass(name, bases, pns)
(Where 'prepare' is the same as above). This way we wouldn't even need to make 'prepare' public, and the new way to create a dynamic type would be:
from operator import build_class cls = build_class(name, bases, ns, **my_kwargs)
No, I think we would want to expose the created namespace directly - that way people can use update(), direct assigment, exec(), eval(), or whatever other mechanism they choose to handle the task of populating the namespace. However, a potentially cleaner way to do that might be offer use an optional callback API rather than exposing a separate public prepare() function. Something like:
def build_class(name, bases=(), kwds=None, eval_body=None): metaclass, ns = _prepare(name, bases, kwds) if eval_body is not None: eval_body(ns) return metaclass(name, bases, ns)
That seems more flexible indeed. I will try to make a patch next week, if that's OK. Daniel
On Mon, Apr 16, 2012 at 5:34 AM, Daniel Urban <urban.dani+py@gmail.com> wrote:
On Sun, Apr 15, 2012 at 13:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
/me pages thoughts from 12 months ago back into brain...
Sorry about that, I planned to do this earlier...
No worries - good to have someone following up on it, since it had completely dropped off my own radar :)
No, I think we would want to expose the created namespace directly - that way people can use update(), direct assigment, exec(), eval(), or whatever other mechanism they choose to handle the task of populating the namespace. However, a potentially cleaner way to do that might be offer use an optional callback API rather than exposing a separate public prepare() function. Something like:
def build_class(name, bases=(), kwds=None, eval_body=None): metaclass, ns = _prepare(name, bases, kwds) if eval_body is not None: eval_body(ns) return metaclass(name, bases, ns)
That seems more flexible indeed. I will try to make a patch next week, if that's OK.
Sure, just create a new tracker issue and assign it to me. You already know better than most what the _prepare() step needs to do :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Apr 16, 2012 at 04:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
Sure, just create a new tracker issue and assign it to me. You already know better than most what the _prepare() step needs to do :)
I've created http://bugs.python.org/issue14588, and attached the first version of a patch. I can't assign it to you, but you're on the nosy list. Thanks, Daniel
(Sorry I'm so late to this discussion.) I think that it's important to take into account the fact that PEP 3115 doesn't require namespaces to implement anything more than __setitem__ and __getitem__ (with the latter not even needing to do anything but raise KeyError). Among other things, this means that .update() is right out as a general-purpose solution to initializing a 3115-compatible class: you have to loop and set items explicitly. So, if we're providing helper functions, there should be a helper that handles this common case by taking the keywords (or perhaps an ordered sequence of pairs) and doing the looping for you. Of course, once you're doing that, you might as well implement it by passing a closure into __build_class__... More below: On Sun, Apr 15, 2012 at 7:48 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yup, I believe that was my main objection to exposing __build_class__ directly. There's no obligation for implementations to build a throwaway function to evaluate a class body.
Thing is, though, if an implementation is dynamic enough to be capable of supporting PEP 3115 *at all* (not to mention standard exec/eval semantics), it's going to have no problem mimicking __build_class__. I mean, to implement PEP 3115 namespaces, you *have* to support exec/eval with arbitrary namespaces. From that, it's only the tiniest of steps to wrapping that exec/eval in a function object to pass to __build_class__. Really, making that function is probably the *least* of the troubles an alternate implementation is going to have with supporting PEP 3115 (by far). Hell, supporting *metaclasses* is the first big hurdle an alternate implementation has to get over, followed by the exec/eval with arbitrary namespaces. Personally, I think __build_class__ should be explicitly exposed and supported, if for no other reason than that it allows one to re-implement old-style __metaclass__ support in 2.x modules that rely on it... and I have a lot of those to port. (Which is why I also think the convenience API for PEP 3115-compatible class creation should actually call __build_class__ itself. That way, if it's been replaced, then the replaced semantics would *also* apply to dynamically-created classes.) IOW, there'd be two functions: one that's basically "call __build_class__", and the other that's "call __build_class__ with a convenience function to inject these values into the prepared dictionary". Having other convenience functions that reimplement lower-level features than __build_class__ (like the prepare thing) sounds like a good idea, but I think we should encourage common cases to just call something that keeps the __setitem__ issue out of the way. Thoughts?
On Sun, Apr 22, 2012 at 12:55 AM, PJ Eby <pje@telecommunity.com> wrote:
(Sorry I'm so late to this discussion.)
I think that it's important to take into account the fact that PEP 3115 doesn't require namespaces to implement anything more than __setitem__ and __getitem__ (with the latter not even needing to do anything but raise KeyError).
Among other things, this means that .update() is right out as a general-purpose solution to initializing a 3115-compatible class: you have to loop and set items explicitly. So, if we're providing helper functions, there should be a helper that handles this common case by taking the keywords (or perhaps an ordered sequence of pairs) and doing the looping for you.
Of course, once you're doing that, you might as well implement it by passing a closure into __build_class__...
Yeah, the "operator.build_class" in the tracker issue ended up looking a whole lot like the signature of CPython's __build_class__. The main difference is that the class body evaluation argument moves to the end and becomes optional in order to bring the first two arguments in line with those of type(). The signature ends up being effectively: def build_class(name, bases=(), kwds={}, exec_body=None): ... Accepting an optional callback that is given the prepared namespace as an argument just makes a lot more sense than either exposing a separate prepare function or using the existing __build_class__ signature directly (which was designed with the compiler in mind, not humans).
Personally, I think __build_class__ should be explicitly exposed and supported, if for no other reason than that it allows one to re-implement old-style __metaclass__ support in 2.x modules that rely on it... and I have a lot of those to port. (Which is why I also think the convenience API for PEP 3115-compatible class creation should actually call __build_class__ itself. That way, if it's been replaced, then the replaced semantics would *also* apply to dynamically-created classes.)
No, we already have one replaceable-per-module PITA like that (i.e. __import__). I don't want to see us add another one.
Having other convenience functions that reimplement lower-level features than __build_class__ (like the prepare thing) sounds like a good idea, but I think we should encourage common cases to just call something that keeps the __setitem__ issue out of the way.
Thoughts?
Agreed on the use of a callback to avoid making too many assumptions about the API provided by the prepared namespace. Definitely *not* agreed on making __build_class__ part of the language spec (or even officially supporting people that decide to replace it with their own alternative in CPython). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Apr 21, 2012 at 11:30 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Apr 22, 2012 at 12:55 AM, PJ Eby <pje@telecommunity.com> wrote:
Personally, I think __build_class__ should be explicitly exposed and supported, if for no other reason than that it allows one to re-implement old-style __metaclass__ support in 2.x modules that rely on it... and I have a lot of those to port. (Which is why I also think the convenience API for PEP 3115-compatible class creation should actually call __build_class__ itself. That way, if it's been replaced, then the replaced semantics would *also* apply to dynamically-created classes.)
No, we already have one replaceable-per-module PITA like that (i.e. __import__). I don't want to see us add another one.
Well, it's more like replacing than adding; __metaclass__ has this job in 2.x. PEP 3115 removed what is (IMO) an important feature: the ability for method-level decorators to affect the class, without needing user-specified metaclasses or class decorators. This is important for e.g. registering methods that are generic functions, without requiring the addition of redundant metaclass or class-decorator statements, and it's something that's possible in 2.x using __metaclass__, but *not* possible under PEP 3115 without hooking __build_class__. Replacing builtins.__build_class__ allows the restoration of __metaclass__ support at the class level, which in turn allows porting 2.x code that uses this facility. To try to be more concrete, here's an example of sorts: class Foo: @decorate(blah, fah) def widget(self, spam): ... If @decorate needs access to the 'Foo' class object, this is not possible under PEP 3115 without adding an explicit metaclass or class decorator to support it. And if you are using such method-level decorators from more than one source, you will have to combine their class decorators or metaclasses in some way to get this to work. Further, if somebody forgets to add the extra metaclass(es) and/or class decorator(s), things will quietly break. However, under 2.x, a straightforward solution is possible (well, to me it's straightforward) : method decorators can replace the class' __metaclass__ and chain to the previous one, if it existed. It's like giving method decorators a chance to *also* be a class decorator. Without some *other* way to do this in 3.x, I don't have much of a choice besides replacing __build_class__ to accomplish this use case.
participants (3)
-
Daniel Urban -
Nick Coghlan -
PJ Eby