dealing with decorators hiding metadata of decorated functions
With the discussion of a possible @decorator to help set the metadata of the decorator to that of what the wrapped function has, I had an idea that I wanted to toss out there (this dicussion stems from a blog post I made: http://sayspy.blogspot.com/2006/03/how-to-handle-object-identity-issues.html). The problem I have with the current practice of setting __name__ and __doc__ (updating __dict__ I have no problem with) is that is destroys metadata on the decorator. This seems kind of bad to destroy metadata for introspection purposes. So how about a __decorates__ attribute that points to what something is decorating? Then help() along with 'inspect' can be changed to look for that attribute, and if it is there, use that for introspection. This way advanced users can still poke at the decorator for information while more common introspection will still point to the decorated function instead of the decorator itself, all without data loss! And on top of it all, it's simple to add support for (obviously the actual code in 'inspect' would be different than what is below =) !:: _help = help def help(obj): while hasattr(obj, "__decorates__"): obj = obj.__decorates__ _help(obj) Worst case is advanced users at the command line will be put out by not being able to directly look at obj.__doc__ for the docstring they care about and such. But this should be a minor issue. But if people don't like that, I do have another proposal to add a __signature__ attribute that would hold an object representing the call signature of a function. This could then be put on a decorator so that introspection on the calling requirements of a decorated function are correct instead of the typical ``*args, **kwargs``. Anyway, what do people think of a __decorates__ attribute? -Brett
"Brett Cannon" <brett@python.org> wrote:
With the discussion of a possible @decorator to help set the metadata of the decorator to that of what the wrapped function has, I had an idea that I wanted to toss out there (this dicussion stems from a blog post I made: http://sayspy.blogspot.com/2006/03/how-to-handle-object-identity-issues.html).
[snip] Edward Loper suggested this way back on September 5, 2004. http://mail.python.org/pipermail/python-dev/2004-September/048626.html I was and continue to be +1 on this, though I would go farther and state, like I did at the time, that one shouldn't copy any of the function attributes, they should come 'free', similar to the way that class attributes are 'free' on subclasses. http://mail.python.org/pipermail/python-dev/2004-September/048631.html What would make this _really_ nice is if one didn't need to do anything manually; that the attribute that pointed to the decorated function/object would be automatically applied - though I realize that this may not be generally possible. - Josiah
Josiah Carlson wrote:
"Brett Cannon" <brett@python.org> wrote:
With the discussion of a possible @decorator to help set the metadata of the decorator to that of what the wrapped function has, I had an idea that I wanted to toss out there (this dicussion stems from a blog post I made: http://sayspy.blogspot.com/2006/03/how-to-handle-object-identity-issues.html).
[snip]
Edward Loper suggested this way back on September 5, 2004.
http://mail.python.org/pipermail/python-dev/2004-September/048626.html
I was and continue to be +1 on this,
+1 here, too. Unlike Brett, though, I have no problem with overwriting __name__ and updating __dict__ unconditionally, and overwriting __doc__ if it hasn't already been set. The first two are needed if we expect "print f" and "f.a" to work properly. The function's name is set on the 'def' line, and I'd be much happier seeing that at all levels of the decorator chain, rather than seeing something like "wrapper", "wrapper", ... "wrapper", "f". Annotating decorators will modify the functions' attributes, and this needs to be visible in the final function's dictionary. If a wrapper doesn't set a docstring explicitly, it makes a lot more sense to me to re-uses the original function's docstring rather than leave it at None. My real interest is that it should be possible to get at all the details of the original function (such as its code object), and the obvious way to do that is with a standard attribute that links to the original.
though I would go farther and state, like I did at the time, that one shouldn't copy any of the function attributes, they should come 'free', similar to the way that class attributes are 'free' on subclasses.
Well, that's the idea behind the decorator decorator - simply put @decorator on your decorator function and it will automatically do the right thing.
http://mail.python.org/pipermail/python-dev/2004-September/048631.html
What would make this _really_ nice is if one didn't need to do anything manually; that the attribute that pointed to the decorated function/object would be automatically applied - though I realize that this may not be generally possible.
A slight problem is that not all decorators will wrap the function they decorate - some will only annotate it. However, here's an idea for the @decorator decorator that would make it pretty much automatic, leaves the docstring alone if the decorator has already set it on the wrapper, and builds up a record of the decorators that are wrapping the the original function: def _link_decorated(decorated, orig, decorator): """Link a decorated function with the original""" decorated.__name__ = orig.__name__ decorated.__dict__.update(orig.__dict__) if decorated.__doc__ is None: decorated.__doc__ = orig.__doc__ decorated.__decorates__ = orig decorated.__decorator__ = decorator def decorator(orig_decorator): """Decorator to create a well-behaved decorator""" # Wrapper function that links a decorated function # to the original function if necessary def wrapper(f): decorated = orig_decorator(f) if decorated is not f: # Link wrapper function to the original _link_decorated(decorated, f, wrapper) return decorated _link_decorated(wrapper, orig_decorator, decorator) return wrapper Decorators that only do annotations aren't recorded because there isn't anywhere to record them. Wrapping decorators, on the other hand, allow the references to both the decorated function and the applied decorator to be stored on the new function object. Cheers, Nick. P.S. Example usage: Py> @decorator ... def annotated(f): ... f.note = 1 ... return f ... Py> @decorator ... def wrapped(f): ... def wrapper(*args, **kwds): ... return f(*args, **kwds) ... return wrapper ... Py> @wrapped ... @annotated ... @wrapped ... @wrapped ... def show(*args, **kwds): ... print args, kwds ... Py> while hasattr(obj, "__decorates__"): ... print obj ... print " Decorates:\t%s" % obj.__decorates__ ... print " Using:\t%s" % obj.__decorator__ ... print " Annotated?:\t%s" % hasattr(obj, "note") ... print ... obj = obj.__decorates__ ... <function show at 0x00AE9D30> Decorates: <function show at 0x00AE9AB0> Using: <function wrapped at 0x00AE9CF0> Annotated?: True <function show at 0x00AE9AB0> Decorates: <function show at 0x00AE9B30> Using: <function wrapped at 0x00AE9CF0> Annotated?: True <function show at 0x00AE9B30> Decorates: <function show at 0x00AE99F0> Using: <function wrapped at 0x00AE9CF0> Annotated?: False -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
On 3/17/06, Nick Coghlan <ncoghlan@gmail.com> wrote:
Josiah Carlson wrote:
"Brett Cannon" <brett@python.org> wrote:
With the discussion of a possible @decorator to help set the metadata of the decorator to that of what the wrapped function has, I had an idea that I wanted to toss out there (this dicussion stems from a blog post I made: http://sayspy.blogspot.com/2006/03/how-to-handle-object-identity-issues.html).
[snip]
Edward Loper suggested this way back on September 5, 2004.
http://mail.python.org/pipermail/python-dev/2004-September/048626.html
I was and continue to be +1 on this,
+1 here, too. Unlike Brett, though, I have no problem with overwriting __name__ and updating __dict__ unconditionally, and overwriting __doc__ if it hasn't already been set.
I have no problem with updating __dict__ since decorators that just do annotations will need to have that information passed forward. But if Josiah's suggestion is taken and decorators are just part of a lookup chain of attributes and they just fall through then this would not be needed.
The first two are needed if we expect "print f" and "f.a" to work properly.
I don't know if having ``print f`` work like that is important. And if it is print can be tweaked to follow __decorates__ properly.
The function's name is set on the 'def' line, and I'd be much happier seeing that at all levels of the decorator chain, rather than seeing something like "wrapper", "wrapper", ... "wrapper", "f". Annotating decorators will modify the functions' attributes, and this needs to be visible in the final function's dictionary.
If a wrapper doesn't set a docstring explicitly, it makes a lot more sense to me to re-uses the original function's docstring rather than leave it at None.
But this is still a loss of information since you will no longer know that the decorator had no docstring.
My real interest is that it should be possible to get at all the details of the original function (such as its code object), and the obvious way to do that is with a standard attribute that links to the original.
Exactly. __decorates__ would provide this. Otherwise the last key piece of information I think missing that decorators cannot copy from the decorated function is the function parameters, and that is where my __signature__ object proposal (I think Philip proposed the main idea originally) steps in if the __decorates__ idea doesn't go anywhere.
though I would go farther and state, like I did at the time, that one shouldn't copy any of the function attributes, they should come 'free', similar to the way that class attributes are 'free' on subclasses.
Well, that's the idea behind the decorator decorator - simply put @decorator on your decorator function and it will automatically do the right thing.
http://mail.python.org/pipermail/python-dev/2004-September/048631.html
What would make this _really_ nice is if one didn't need to do anything manually; that the attribute that pointed to the decorated function/object would be automatically applied - though I realize that this may not be generally possible.
A slight problem is that not all decorators will wrap the function they decorate - some will only annotate it.
However, here's an idea for the @decorator decorator that would make it pretty much automatic, leaves the docstring alone if the decorator has already set it on the wrapper, and builds up a record of the decorators that are wrapping the the original function:
def _link_decorated(decorated, orig, decorator): """Link a decorated function with the original""" decorated.__name__ = orig.__name__ decorated.__dict__.update(orig.__dict__) if decorated.__doc__ is None: decorated.__doc__ = orig.__doc__ decorated.__decorates__ = orig decorated.__decorator__ = decorator
def decorator(orig_decorator): """Decorator to create a well-behaved decorator""" # Wrapper function that links a decorated function # to the original function if necessary def wrapper(f): decorated = orig_decorator(f) if decorated is not f: # Link wrapper function to the original _link_decorated(decorated, f, wrapper) return decorated _link_decorated(wrapper, orig_decorator, decorator) return wrapper
Decorators that only do annotations aren't recorded because there isn't anywhere to record them. Wrapping decorators, on the other hand, allow the references to both the decorated function and the applied decorator to be stored on the new function object.
If you do the copying of data and provide a __signature__ object you have 90% of the metadata one would want for introspection on a decorated function, so I don't know if __decorates__ will be that important in that situation (but one extra pointer to the function object is not that expensive, so it can still be provided very cheaply). I guess we need to decide if we want to promote the copying of metadata from the decorated function into the decorator or not. If we do support copying the metadata, then should we provide a __signature__ object as well to help with that or not. __decorates__ is an innocuous suggestion that it should probably be promoted regardless of what we end up suggesting for use. The real difference will be whether 'inspect' and friends get tweaked to follow __decorates__ or just to use the metadata on the decorator. -Brett
Cheers, Nick.
P.S. Example usage:
Py> @decorator ... def annotated(f): ... f.note = 1 ... return f ... Py> @decorator ... def wrapped(f): ... def wrapper(*args, **kwds): ... return f(*args, **kwds) ... return wrapper ...
Py> @wrapped ... @annotated ... @wrapped ... @wrapped ... def show(*args, **kwds): ... print args, kwds ...
Py> while hasattr(obj, "__decorates__"): ... print obj ... print " Decorates:\t%s" % obj.__decorates__ ... print " Using:\t%s" % obj.__decorator__ ... print " Annotated?:\t%s" % hasattr(obj, "note") ... print ... obj = obj.__decorates__ ... <function show at 0x00AE9D30> Decorates: <function show at 0x00AE9AB0> Using: <function wrapped at 0x00AE9CF0> Annotated?: True
<function show at 0x00AE9AB0> Decorates: <function show at 0x00AE9B30> Using: <function wrapped at 0x00AE9CF0> Annotated?: True
<function show at 0x00AE9B30> Decorates: <function show at 0x00AE99F0> Using: <function wrapped at 0x00AE9CF0> Annotated?: False
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
On Sun, Mar 19, 2006, Brett Cannon wrote:
I guess we need to decide if we want to promote the copying of metadata from the decorated function into the decorator or not.
Short answer: absolutely yes I had to deal with a related issue recently, where embedded doctests no longer worked on the decorated function. There were two separate problems: the first was non-copying of __doc__, which was easy enough to fix; the second (which I still haven't fixed) is the fact that __name__ wasn't set, so doctest didn't auto-find the function. (The decorator came from a different module.) (This was Python 2.2/2.3, so decorators weren't actually involved, but I think it's the same problem.) I don't understand decorators well enough to propose a good solution given the other constraints, but it seems clear to me that we need a good solution of some kind. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "19. A language that doesn't affect the way you think about programming, is not worth knowing." --Alan Perlis
participants (4)
-
Aahz -
Brett Cannon -
Josiah Carlson -
Nick Coghlan