
Hello everybody, My name is Christophe Schlick, from the University of Bordeaux, France. I've been using Python for years, but it happens that this is my first post on a Python-related mailing list. For the first time, I feel that I may have some topic interesting enough for the Python community, so I would be very happy to get any kind of feeback (positive or negative) on it. Thanks in advance... The goal of this post is to propose a new syntax for defining decorators in Python. I would like to fully explain the rationale and the benefits of this proposal, as I guess that the discussion might be more fruitful if all details are written down once for all. However this has led me to a very long post (a kind of informal PEP) that was blocked by the moderators. So, following their recommendation, I've divided the text in three successive parts: (1) the rationale, (2) the proposal, (3) the proposed implementation and additional questions/remarks. For better clarity of the description, I will call the proposed syntax as "new-style decorators" (NSD, for short) and rename the classic syntax as "old-style decorators" (OSD), following the terms used some years ago with the (re)definition of classes. By the way, the introducing process for NSD shares many aspects with the process used for introducing new-style classes, including the following features: * No existing syntax is broken: the only thing required to create a new-style decorator is to decorate itself by a newly-introduced decorator called... "decorator" (well, this sentence is less recursive than it might appear at the first reading). * Every thing that can be done with OSD is possible with NSD, but NSD offer additional more user-friendly features. * NSD can peacefully live together with OSD in the same code. An NSD may even decorate an OSD (and vice-versa), however some properties of the NSD are lost with such a combination. -------------------------------------------------- 1 - Why bother with a new syntax? To explain what I don't like with the current syntax of decorators, let me take the example of a basic decorator (called 'old_style_repeat_fix') that simply repeats 3 times its undecorated function, and adds some tracing to the standard output. Here is the code: #--- def old_style_repeat_fix(func): """docstring for decorating function""" # @wraps(func) def dummy_func_name_never_used(*args, **keys): """docstring for decorated function""" print "apply 'old_style_repeat_fix' on %r" % func.__name__ for loop in range(3): func(*args, **keys) return dummy_func_name_never_used #--- Even if such code snippets have become quite usual since the introduction of decorators in Python 2.2, many people have argued (and I am obviously one of them) that the decorator syntax is a bit cumbersome. First, it imposes the use of nested functions, which often reduces readability by moving the function signature and docstring too far from the corresponding code. Second, as anonymous lambdas expressions can usually not be employed for decorating functions, the programmer has no other choice than to create a dummy function name (only used for one single 'return' statement), which is never a good coding principle, whatever the programming language. Once you have tried to teach decorators to a bunch of students, you really understand how much this syntax leverages the difficulty to grab the idea. The situation is even worse when the decorator needs some arguments: let's create an extended decorator (called 'old_style_repeat_var) that includes an integer 'n' to control the number of iterations, and a boolean 'trace' to control the tracing behavior. Here is the code: #--- def old_style_repeat_var(n=3, trace=True): """docstring for decorating function""" def dummy_deco_name_never_used(func): """docstring never used""" # @wraps(func) def dummy_func_name_never_used(*args, **keys): """docstring for decorated function""" if trace: print "apply 'old_style_repeat_var' on %r" % func.__name__ for loop in range(n): func(*args, **keys) return dummy_func_name_never_used return dummy_deco_name_never_used #--- This time a two-level function nesting is required and the code needs two dummy names for these two nested functions. Note that the docstring of the middle nested function is even totally invisible for introspection tools. So whether you like nested functions or not, there is some evidence here that the current syntax is somehow suboptimal. Another drawback of OSD is that they do not gently collaborate with introspection and documentation tools. For instance, let's apply our decorator on a silly 'test' function: #--- @old_style_repeat_var(n=5) # 'trace' keeps its default value def test(first=0, last=0): """docstring for undecorated function""" print "test: first=%s last=%s" % (first, last) #--- Now, if we try 'help' on it, we get the following answer: #---
which means that neither the name, nor the docstring, nor the signature of the 'test' function are correct. Things are a little better when using the 'wraps' function from the standard 'functools' module (simply uncomment the line '@wraps(func)' in the code of 'old_style_repeat_var'): #---
'@wraps(func)' copies the name and the docstring from the undecorated function to the decorated one, in order to get some useful piece of information when using 'help'. However, the signature of the function still comes from the decorated function, not the genuine one. The reason is that signature copying is not an easy process. The only solution is to inspect the undecorated function and then use 'exec' to generate a wrapper with a correct signature. This is basically what is done in the 'decorator' package (available at PyPI) written by Michele Simionato. There has been a lengthy discussion in python-dev (in 2009 I guess, but I can't find the archive right now) whether to include or not this package in the standard library. As far as I know, there is currently no clear consensus whether this is a good idea or not, because there has always been a mixed feeling from the community about transparent copy from the undecorated to the decorated function (even about the 'wraps' function): on one hand, transparent copy is cool for immediate help, for automatic documentation and for introspection tools, but on the other hand, it totally hides the decorating process which is not always what is wanted... or needed. The syntax for NSD presented in this proposal tries to improve this situation by offering two desirable features, according to the Zen of Python: * "flat is better than nested": no nested functions with dummy names are required, even when parameters are passed to the decorator; only one single decorating function has to be written by the programmer, whatever the kind of decorator. * "explicit is better than implicit": introspection of a decorated function explicitely reveals the decoration process, and allows one to get the name/signature/docstring not only for the corresponding undecorated function, but also for any number of chained decorators that have been applied on it. ------ to be continued in Part 2... CS

Christophe Schlick wrote:
The goal of this post is to propose a new syntax for defining decorators in Python.
Too long, did read it. Here's the summary so others don't have to: "The current syntax for writing decorator functions is cumbersome, has too much boilerplate, and doesn't play well with introspection tools. I've got some great new syntax for writing decorators, but you'll have to wait to find out what it is." You've obviously spent a lot of effort thinking this through, but this shouldn't be a mystery novel where you are trying to keep Who Done It secret until the end of the book. You talk a lot about new decorator syntax, but after reading > 150 lines, I still have no idea what it is, how much work it will require, and how disruptive it will be. That doesn't make me very enthusiastic about the idea. Scientific papers and reports often have an executive summary at the very beginning: one or two paragraphs that summarize the report without all the details. Perhaps you should do the same? As for the rationale given in this post, I'm not convinced there's actually a problem that needs solving. The "problems" you list seem pretty minor to me: e.g. so what if you have to name the inner function? Admittedly, that the function signature is lost when using decorators *is* a pretty annoying gotcha, but surely we don't need new *syntax* to fix that. New *functionality* in functools.wraps perhaps? Oh, one thought that comes to me... decorators are a special case of factory functions. Does your proposed syntax (whatever it is!) treat decorator functions as a special case, or does it apply to general factory functions as well? -- Steven

I am really sorry for splitting the text. My first post was in one piece but got blocked by the moderators. I didn't want to create some ridiculous suspense at the end of the first part. Here is the second part... --- Part 2 - The new-style syntax for decorators: Here is the code of the same decorators as in Part 1 above, but the proposed syntax. The first decorator ('new_style_repeat_fix') is created without parameters, while the second one ('new_style_repeat_var') uses arguments 'n' and 'trace' with the same role as previously: #--- @decorator def new_style_repeat_fix(self, *args, **keys): """docstring for decorating function""" print "apply %r on %r" % (self.deco.__name__, self.func.__name__) for n in range(3): self.func(*args, **keys) @decorator(n=3, trace=True) def new_style_repeat_var(self, *args, **keys): """docstring for decorating function""" if self.trace: print "apply %r on %r" % (self.deco.__name__, self.func.__name__) for n in range(self.n): self.func(*args, **keys) #--- When examining the new-style syntax, one can notice that there are basically four characteristics that distinguish NSD from OSD: * Each NSD is decorated by a newly-introduced callable class called 'decorator' (code provided in Part 3) by using one of two possible decorating forms. The first form '@decorator' is employed for decorators that do not need any parameter. The second form '@decorator(arg1=val1, arg2=val2...)' is employed to specify a sequence of named arguments (combined with their default values) that are passed to the decorator using the standard notation for keyword arguments. * The role of the 'decorator' class is to generate the decorated function (i.e. the inner nested function with the classic OSD syntax) and to broadcast it to the decorating function as its first argument 'self'. When the second decorating form is used, all keyword arguments used in '@decorator(...)' are automatically injected as meta-attributes in this decorated function 'self'. In the example above, the two decorator arguments 'n' and 'trace' are available within the code of 'new_style_repeat_var' as 'self.n' and 'self.trace' respectively. This mechanism avoids one level of nested functions used in standard OSD. * In addition to these decorator arguments, the decorating process also injects two other meta-attributes in the decorated function: 'self.deco' represents a reference to the decorating function, while 'self.func' represents a reference to the undecorated function. If there are several chained decorators, the mechanism is made recursive (more on this later). Note that this implies a slight name restriction: neither 'deco' nor 'func' can be used as the name of a parameter passed to the decorator, as this would generate collision in the corresponding namespace. An alternative might be to consider the two references as "special" attributes and rename them as 'self.__deco__' and 'self.__func__' respectively. I have no clear opinion about the pros/cons of the two alternatives. * Finally, note that each NSD has the same 3-argument signature: (self, *args, **keys). The first argument 'self' has been explained above. The two others 'args' and 'keys' respectively represent the set of positional and keyword arguments, as usual. However, all the values in either 'args' or 'keys' are not meant to be used by the decorating function, but always directly passed to the undecorated function. This means that the statement 'self.func(*args, **keys)' will always appear somewhere in the code of an NSD. Following this mechanism in the decorating function avoids the other level of nested functions used in standard OSD, and guarantees that flat functions are always sufficient. Once the NSD have been defined with the new syntax, they can be used to decorate functions using the standard @-notation, either for single or multiple decoration. For instance: #--- @new_style_repeat_fix def testA(first=0, last=0): """docstring for undecorated function""" print "testA: first=%s last=%s" % (first, last) @new_style_repeat_var(n=5) # 'n' is changed, 'trace' keeps default value def testB(first=0, last=0): """docstring for undecorated function""" print "testB: first=%s last=%s" % (first, last) @new_style_repeat_var # both 'n' and 'trace' keep their default values @new_style_repeat_fix @new_style_repeat_var(n=5, trace=False) # both 'n' and 'trace' are changed def testC(first=0, last=0): """docstring for undecorated function""" print "testC: first=%s last=%s" % (first, last) #--- When applying a decorator without arguments, or when *all* its arguments use their default values, the parenthesis after the decorator name may be dropped. In other words, '@mydeco' and '@mydeco()' are totally equivalent, whether 'mydeco' takes arguments or not. This solves a non-symmetric behavior of standard OSD that has always bothered me: '@old_style_repeat_fix' works but '@old_style_repeat_fix()' does not, and inversely '@old_style_repeat_var()' works but '@old_style_repeat_var' does not. Note also that in the case of chained decorators, each decoration level stores its own set of parameters, so there is no conflict when applying the same decorator several times on the same function, as done with 'new_style_repeat_var' on 'testC'. Now let's play a bit with some introspection tools: #---
testA <function <deco>testA...>
testB <function <deco>testB...>
To explicitely expose the decoration process, a '<deco>' substring is added as a prefix to the '__name__' attribute for each decorated function (more precisely, there is one '<deco>' for each level of decoration, as can be seen with 'testC'). So, each time a '<deco>' prefix is encountered, the user knows that the reference to the corresponding undecorated function (resp. decorating function) is available through the meta-attribute '.func' (resp. '.deco'). When calling 'help' on a decorated function, this principle is clearly displayed, and the user can thus easily obtain useful information, including correct name/signature/docstring: #---
testA.func, testA.deco (<function testA...>, <function new_style_repeat_fix...>)
In the case of chained decorators, the same principle holds recursively. As can be seen in the example below, all information relative to a multi-decorated function (including all decorator arguments used at any decoration level) can be easily fetched by successive applications of the '.func' suffix: #---
testC.func, testC.deco, testC.n, testC.trace (<function <deco><deco>testC...>, <function new_style_repeat_var...>, 3, True)
testC.func.func, testC.func.deco (<function <deco>testC...>, <function new_style_repeat_fix...>)
testC.func.func.func, testC.func.func.deco, testC.func.func.n, testC.func.func.trace (<function testC...>, <function new_style_repeat_var...>, 5, False)
------ to be continued in Part 3... CS

Part 3 - Implementation and additional remarks/questions I've implemented the idea of NSD about nine months ago and have used them successfully in many different situations. My first implementation was terribly messy but at least it did the job. Last week, while thinking about refactoring, I realized that the whole process can be easily implemented as a state machine. This has led me to an extremely compact implementation (about 30 Python statements), where a callable class repeatedly returns reference to itself, until it gets all the required parameters to generate and return the decorated function. Here is the code, so that you can experiment with it, if you wish: #--- class decorator(object): """apply a 'new-style' decorator to a function""" def __init__(self, deco=None, **keys): # either get reference or default parameters for decorating function self.deco = deco; self.keys = keys; self.stack = list() def __call__(self, func=None, **keys): if self.deco is None: # get reference for decorating function self.deco = func; return self elif func is None: # get modified parameters of decorating function self.stack.append(keys); return self else: # get undecorated function and generate decorated function deco = node = lambda *args, **keys: self.deco(deco, *args, **keys) deco.func = func; deco.deco = self.deco; deco.__dict__.update(self.keys) if self.stack: deco.__dict__.update(self.stack.pop()) head = '<deco>'; deco.__name__ = name = head + func.__name__ level = name.count(head); offset = len(head)*level; tail = '.func'*level doc = "use help(%s) to get genuine help" % (name[offset:] + tail) while hasattr(node, 'func'): node.__doc__ = doc; node = node.func return deco #--- The simplicity of the new implementation has convinced me that it might be useful to share this idea and write a proposal in order to get some feedback from the community. As said in the introduction, this is my first post to python-ideas, so I'm not sure about the correct process to follow. I've got plenty of questions anyway: * Is the idea interesting enough to deserve consideration for possible inclusion in the language? If yes, should I transform this proposal into a PEP, or should there first be some pre-PEP discussion here (or maybe in python-dev)? * Are there some pitfalls involved with the use of NSD that I haven't seen? Or are there additional desirable elements that could be easily included? * After having read this proposal, has anybody some suggestion for alternative syntax that offer similar features? * There are some advanced features offered by the new syntax (such as meta-decorator, or whatever you call them), which seem to be powerful but are less stable than the elements presented here. I did not detail this kind of stuff because I consider that it is likely to create some noise in the discussion, but maybe not? Thanks for any opinion, CS

Christophe Schlick wrote:
Decorators were one of the biggest successes in recent years, so we would be foolish to dismiss the idea of simplifying them out of hand. But I think you are doing yourself a disservice by referring to this proposal as "new syntax". Normally when people talk about syntax, they are referring to language syntax (i.e. a change to the Python interpreter), and we're pretty conservative about adding new syntax. It seems to me that you're talking about a new idiom for building decorator functions, not new syntax. I would suggest you also publish this decorator-builder recipe on ActiveState's Python cookbook, and see if you get much interest there. It might also help to post a link to your recipe to python-list@python.org. You certainly should do those things before going to python-dev.
Have you timed the decorated function using new and old style? If you decorator a function with (say) 5 arguments, is there any performance hit to your NSD? Do you have any tests for it? E.g. unit tests, regression tests? Your code looks opaque and complicated to me, I would want to see a good test suite before even considering using it in production code, let alone in the standard library. -- Steven

On Tue, Apr 26, 2011 at 8:08 PM, Steven D'Aprano <steve@pearwood.info> wrote:
You are right. I've mostly used the expression "new-style decorators" (which corresponds to the idea of "new idiom" that you propose) but I guess that there are still a couple of unwanted "new syntax" in the post. At the beginning of the post, I say that "no existing syntax is broken", so the proposal does not change anything for the interpreter. Moreover, the old idiom can be used in combination with the new one, so no existing code can be broken by the proposal. CS

On Tue, Apr 26, 2011 at 8:08 PM, Steven D'Aprano <steve@pearwood.info> wrote:
OK, I'm going to try that. Thanks.
Intuitively I would say the the only performance hit would comme from the fact that the decorator arguments are accessed via self.__dict__ in NSD, while there are available as locals with OSD. I've made some quick 'timeit' tests. I don't know if this is the kind of timing you thought about: #--- from timeit import Timer from decorator import decorator # OSD def old_add_args(a=1, b=2, c=3, d=4, e=5): def dummy1(func): def dummy2(*args, **keys): return a + b + c + d + e + func(*args, **keys) return dummy2 return dummy1 # NSD @decorator(a=1, b=2, c=3, d=4, e=5) def new_add_args(self, *args, **keys): return self.a + self.b + self.c + self.d + self.e + self.func(*args, **keys) # Apply OSD @old_add_args() def old_test(*args, **keys): return sum(*args) # Apply NSD @new_add_args() def new_test(*args, **keys): return sum(*args) # Gentle case: the evaluation of the function is rather long compared to the time # needed to fetch the 5 decorator args old_time = Timer('old_test(range(999))', 'from __main__ import old_test').timeit() new_time = Timer('new_test(range(999))', 'from __main__ import new_test').timeit() print "Gentle: old = %.3f new = %.3f" % (old_time, new_time) # Worst case: the evaluation of the function is negligible compared to the time # needed to get the 5 decorators args. old_time = Timer('old_test(range(1))', 'from __main__ import old_test').timeit() new_time = Timer('new_test(range(1))', 'from __main__ import new_test').timeit() print "Worst: old = %.3f new = %.3f" % (old_time, new_time) #--- Here are the timings obtained on my notebook: Gentle: old = 45.983 new = 46.377 Worst: old = 4.043 new = 5.127 which seems to confirm that the overhead mainly comes from the 'self.xxx' fetch, and it pretty negligible when heavy computation is performed. CS

Just before I go to bed, here some alternative implementations: Usage: ------ # for argument-less decorators: @decorator def my_deco(func,*func_args,**func_kwargs): pass # of course regular arguments can also be declared (same for all funcs below) @my_deco def func(*func_args,**func_kwargs): pass # for decorators with arguments: # even when there are no default arguments function-call parenthesis are needed @decorator_with_args('foo',bar='baz') def my_deco(func,deco_args,deco_kwargs,*func_args,**func_kwargs): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass # alternative version where the decorator arguments are expanded: # `...` is a placeholder for the arguments of the decorator in regular argument syntax. # This way the decorator arguments can be declared inline and no deco_(kw)args or self.* # is needed. Also decorator arguments are not decoupled from their default values this way. @decorator_with_expanded_args def my_deco(func,...,*func_args,**func_kwargs): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass Implementation: --------------- from types import FunctionType, ClassType from functools import wraps def decorator(deco): @wraps(deco) def _deco(func): @wraps(func) def _f(*args,**kwargs): return deco(func,*args,**kwargs) return _f return _deco def decorator_with_args(*deco_default_args,**deco_default_kwargs): def _deco_deco_deco(deco): @wraps(deco) def _deco_deco(*deco_args,**deco_kwargs): if len(deco_args) < len(deco_default_args): deco_args = deco_args+deco_default_args[len(deco_args):] merged_deco_kwargs = dict(deco_default_kwargs) merged_deco_kwargs.update(deco_kwargs) del deco_kwargs def _deco(func): @wraps(func) def _f(*args,**kwargs): return deco( func,deco_args,merged_deco_kwargs,*args,**kwargs) return _f return _deco return _deco_deco return _deco_deco_deco def decorator_with_expanded_args(deco): if isinstance(deco, FunctionType): co = deco.func_code deco_name = deco.func_name arg_names = list(co.co_varnames[0:co.co_argcount]) elif isinstance(deco, ClassType): co = deco.__init__.func_code deco_name = deco.__name__ arg_names = list(co.co_varnames[1:co.co_argcount]) elif hasattr(deco, '__call__'): co = deco.__call__.func_code deco_name = type(deco).__name__ arg_names = list(co.co_varnames[0:co.co_argcount]) else: raise TypeError('not a decorateable object') if not arg_names: raise TypeError('decorator function needs a func argument') del co del arg_names[0] min_argcount = len(arg_names) if deco.func_defaults: min_argcount -= len(deco.func_defaults) @wraps(deco) def _deco_deco(*args,**kwargs): deco_args = list(args) n = len(deco_args) if n < len(arg_names): i = n - min_argcount for arg in arg_names[n:]: if arg in kwargs: deco_args.append(kwargs.pop(arg)) elif i < 0: raise TypeError( '%s() takes at least %d positional ' + 'arguments (%d given)' % (deco_name, min_argcount, len(deco_args))) else: deco_args.append(deco.func_defaults[i]) i += 1 if kwargs: arg = kwargs.keys()[0] if arg in arg_names: raise TypeError( "%s() got multiple values for keyword argument '%s'" % (deco_name, arg)) else: raise TypeError("%s() got an unexpected keyword argument '%s'" % (deco_name, arg)) deco_args = tuple(deco_args) def _deco(func): @wraps(func) def _f(*args,**kwargs): return deco(func,*(deco_args+args),**kwargs) return _f return _deco return _deco_deco What do you think? -panzi

On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick@gmail.com>wrote:
I'm sure it's not exactly as you envisioned, but the decorator package provides roughly what you're describing: http://micheles.googlecode.com/hg/decorator/documentation.html#decorator-is-... -- Daniel Stutzbach

On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick@gmail.com> wrote: <snip>
I'm personally not a fan of magic such as having both @decorator and @decorator(...) work.
What if my decorator's parameters don't all have default values? What if I don't want people to have to use keyword arguments when using my decorator? What if my decorator accepts extra positional arguments (i.e. *args)? Cheers, Chris

On Wed, Apr 27, 2011 at 12:30 AM, Chris Rebert <pyideas@rebertia.com> wrote:
Well, you may simply use @decorator(), if your decorator does not need arguments. Actually the magic does not come from the new 'decorator' class, but directly from the state machine used to parse the @-syntax in Python. I am not responsible for that.
The idea behind the proposal is to reduce the boilerplate for most of the standard patterns of decorators. One element of that reduction is to automatically transform the decorator arguments as attributes of the decorated function. To do this, the attributes have to get individual names, that's why I've proposed the keyword argument syntax. However, it is actually possible to implement the same idea by letting the decorator use positional arguments, which are then combined into a single tuple attribute 'self.args' available for the decorating function. The code of the 'decorator' class would simply be a bit longer, but there is no specific difficulty here. CS

On 4/26/2011 11:05 AM, Christophe Schlick wrote: I got interrupted in responding this, and you have now posted parts 2 and 3, so I will snip and revise a bit.
a new syntax for defining decorators in Python.
There is no special syntax for defining decorators -- just normal nested function or class definition syntax. To put it another way, Python does not have decorator objects. A decorator is simply a callable (function, class, or class instance with __call__ method) applied to a object (function or class) with the @deco syntax (before the def/class statement) instead of the normal call syntax (after the def/class statement). Decorators return either the original object (usually modified) or another object that is usually, but not necessarily, of the same kind as the input. As Stephen noted, syntax is what is defined in the Language reference. Code patterns are what are embodied in the stdlib (or pypi or the Python cookbook or other example repositories). What you are actually proposing is a meta-decorator (class) whose instances can be used as decorators because the class has a __call__ instance method. This sort of thing is a known alternative to the nested function pattern.
programmer has no other choice than to create a dummy function name
Many of us consider dummy names a bogus problem. I recommend you skip this argument. In any case, this, like the other 'problems' you describe for nested functions, has nothing in particular with their usage as decorators.
(only used for one single 'return' statement), which is never a good coding principle, whatever the programming language.
This strikes me as a bogus rule: a single use of a local name is quite common, and not just in Python. I recommend leaving this also out of your problem list. Stick with the two real problems. 1. The double or triple nested function pattern has a lot of boilerplate and can be difficult to learn. Hiding boilerplate in a class makes the use pattern simpler and easier to learn. This is a real benefit. One of the three major benefits of writing a generator function versus an equivalent iterator class is that is hides the boilerplate required for the latter. Similarly, for loops hide the boilerplate required for an equivalent while loop. 2. Introspection (more comments below), which your class also addresses.
Actually, this is the docstring for the decorator making function.
This time a two-level function nesting is required and the code needs two dummy names for these two nested functions.
'deco' and 'wrapper' work for me. But I agree that this is a bit confusing. But I think that is somewhat inherent in calling a decorator-maker f1 to return decorator f2 that returns wrapper f3 that wraps the original function f.
Note that the docstring of the middle nested function is even totally invisible for introspection tools.
Not true. To see the docstring of a dynamically created temporary function, you have to either dynamically create it or dig inside the function that creates it to find the constant string:
old_style_repeat_var().__doc__ 'docstring never used'
old_style_repeat_var.__code__.co_consts[1].co_consts[0] 'docstring never used'
But I am not sure why you would want to see it or even have one.
The 'problems' of nested defs has nothing to do with decorators in particular. Functional programmers use them all the time.
Only because you commented out @wraps. Again, this is not a problem of the @decorator syntax but of *all* wrapping callables. Functools.partial has the same 'problem'.
I am not sure what you mean. If the two signatures are different, then one must use the signature of the wrapper when calling it, not the signature of the wrappee, which is perhaps what you mean by 'the genuine one'. The problem of generic wrappers having generic signatures (*args, **kwds) is endemic to using generic wrappers instead of special case wrappers.
reason is that signature copying is not an easy process.
True if you want to do it generically. Copying with modification, as functools.partial would have to do, is even worse.
The other solution is to not use a generic wrappers with generic signatures but to write specific wrappers with the actual signature, which people did, for instance, before functools and partial() were added to Python. There have been proposals but no consensus on a decorator or decolib module for the stdlib. I second the other recommendations to make your proposal available on the cookbook site, etc. -- Terry Jan Reedy

I thought about this problem for I wile and I came up with this @decorator: Usage: ------
Or a more specific example:
This fully supports *args and **kwargs besides regular arguments for the decorator and the decorated function. By that I mean func_args already contains the filled in default values of func as well as any regular arguments that where passed as keyword argument and argument passing errors are handled (e.g. passing an argument as positional and keyword argument). Error handling example:
Traceback (most recent call last): File "<pyshell#114>", line 1, in <module> @deco(1,2,3) File "<pyshell#96>", line 22, in _deco_deco deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs) TypeError: deco() takes at most 2 arguments (3 given) Or:
foo(5,6,y=33)
Traceback (most recent call last): File "<pyshell#112>", line 1, in <module> foo(5,6,y=33) File "<pyshell#96>", line 27, in _f func_args, func_kwargs = apply_func_args(*func_args, **func_kwargs) TypeError: foo() got multiple values for keyword argument 'y' Of course that always needs function call parenthesis on the decorator, even if the decorator does not take any arguments. Maybe it could be extended that in this case a more simple decorator mechanism is used. A decorator-decorator for decorators without arguments would be very simple (see end of mail). Implementation: --------------- from types import FunctionType, ClassType from functools import wraps def inspect_callable(func): """-> (arg_names, co_flags, func_defaults, func_name)""" return _inspect_callable(func,set()) def _inspect_callable(func,visited): if func in visited: raise TypeError("'%s' object is not callable" % type(func).__name__) visited.add(func) if isinstance(func, FunctionType): co = func.func_code func_name = func.__name__ arg_names = list(co.co_varnames[0:co.co_argcount]) defaults = func.func_defaults flags = co.co_flags elif isinstance(func, ClassType): func_name = func.__name__ arg_names, flags, defaults, member_name = _inspect_callable(func.__init__,visited) if arg_names: del arg_names[0] elif hasattr(func, '__call__'): func_name = '<%s object at 0x%x>' % (type(func).__name__, id(func)) arg_names, flags, defaults, member_name = _inspect_callable(func.__call__,visited) else: raise TypeError("'%s' object is not callable" % type(func).__name__) return arg_names, flags, defaults, func_name FUNC_ARGS = 0x04 FUNC_KWARGS = 0x08 FUNC_GEN = 0x20 # this function should probably be reimplemented in C: def args_applyer(arg_names,flags=0,defaults=None,func_name=None): """-> f(args..., [*varargs], [**kwargs]) -> ((args...)+varargs, kwargs)""" all_args = list(arg_names) if arg_names: body = ['(',','.join(arg_names),')'] else: body = [] if flags & FUNC_ARGS: args_name = '_args' i = 0 while args_name in arg_names: args_name = '_args'+i i += 1 all_args.append('*'+args_name) if arg_names: body.append('+') body.append(args_name) elif not arg_names: body.append('()') body.append(',') if flags & FUNC_KWARGS: kwargs_name = '_kwargs' i = 0 while kwargs_name in arg_names: kwargs_name = '_kwargs'+i i += 1 all_args.append('**'+kwargs_name) body.append(kwargs_name) else: body.append('{}') if func_name: apply_args = named_lambda(func_name,all_args,''.join(body)) else: apply_args = eval('lambda %s: (%s)' % (','.join(all_args), ''.join(body))) if defaults: apply_args.func_defaults = defaults return apply_args def named_lambda(name,args,body): code = 'def _named_lambda():\n\tdef %s(%s):\n\t\treturn %s\n\treturn %s' % ( name, ','.join(args), body, name) del name, args, body exec(code) return _named_lambda() # begin helper functions (not used by this module but might be handy for decorator developers) def args_applyer_for(func): return args_applyer(*inspect_callable(func)) def apply_args(args,kwargs,arg_names,flags=0,defaults=None,func_name=None): return args_applyer(arg_names,flags,defaults,func_name)(*args,**kwargs) def apply_args_for(func,args,kwargs): return args_applyer(*inspect_callable(func))(*args,**kwargs) # end helper functions def decorator(deco): """deco(func,func_args,func_kwargs,deco_args...) @decorator def my_deco(func,func_args,func_kwargs,deco_args...): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass """ arg_names, flags, defaults, deco_name = inspect_callable(deco) if flags & FUNC_ARGS == 0: if len(arg_names) < 3: raise TypeError('decorator functions need at least 3 ' + 'arguments (func, func_args, func_kwargs)') del arg_names[0:3] apply_deco_args = args_applyer(arg_names,flags,defaults,deco_name) del flags, defaults @wraps(deco) def _deco_deco(*deco_args,**deco_kwargs): deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*func_args,**func_kwargs): func_args, func_kwargs = apply_func_args(*func_args, **func_kwargs) return deco(func,func_args,func_kwargs,*deco_args,**deco_kwargs) return _f return _deco return _deco_deco def simple_decorator(deco): """deco(func,func_args,func_kwargs) @simple_decorator def my_deco(func,func_args,func_kwargs): pass @my_deco def func(*func_args,**func_kwargs): pass """ @wraps(deco) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*args,**kwargs): return deco(func,*apply_func_args(*args,**kwargs)) return _f return _deco

Maybe even better, decorator and simple_decorator in one: def decorator(deco): """deco(func,func_args,func_kwargs,deco_args...) @decorator def my_deco(func,func_args,func_kwargs,deco_args...): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass @decorator def my_deco2(func,func_args,func_kwargs): pass @my_deco2 def func2(*func_args,**func_kwargs): pass @decorator def my_deco3(func,func_args,func_kwargs,x=1): pass @my_deco3() def func3(*func_args,**func_kwargs): pass """ arg_names, flags, defaults, deco_name = inspect_callable(deco) if flags & FUNC_ARGS == 0: if len(arg_names) < 3: raise TypeError('decorator functions need at least 3 ' + 'arguments (func, func_args, func_kwargs)') del arg_names[0:3] if not arg_names and flags & (FUNC_ARGS | FUNC_KWARGS) == 0: # argument-less decorator del arg_names, flags, defaults, deco_name @wraps(deco) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*args,**kwargs): return deco(func,*apply_func_args(*args,**kwargs)) return _f return _deco else: apply_deco_args = args_applyer(arg_names,flags,defaults,deco_name) del arg_names, flags, defaults, deco_name @wraps(deco) def _deco_deco(*deco_args,**deco_kwargs): deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*func_args,**func_kwargs): func_args, func_kwargs = apply_func_args( *func_args, **func_kwargs) return deco(func,func_args,func_kwargs, *deco_args,**deco_kwargs) return _f return _deco return _deco_deco

Christophe Schlick wrote:
The goal of this post is to propose a new syntax for defining decorators in Python.
Too long, did read it. Here's the summary so others don't have to: "The current syntax for writing decorator functions is cumbersome, has too much boilerplate, and doesn't play well with introspection tools. I've got some great new syntax for writing decorators, but you'll have to wait to find out what it is." You've obviously spent a lot of effort thinking this through, but this shouldn't be a mystery novel where you are trying to keep Who Done It secret until the end of the book. You talk a lot about new decorator syntax, but after reading > 150 lines, I still have no idea what it is, how much work it will require, and how disruptive it will be. That doesn't make me very enthusiastic about the idea. Scientific papers and reports often have an executive summary at the very beginning: one or two paragraphs that summarize the report without all the details. Perhaps you should do the same? As for the rationale given in this post, I'm not convinced there's actually a problem that needs solving. The "problems" you list seem pretty minor to me: e.g. so what if you have to name the inner function? Admittedly, that the function signature is lost when using decorators *is* a pretty annoying gotcha, but surely we don't need new *syntax* to fix that. New *functionality* in functools.wraps perhaps? Oh, one thought that comes to me... decorators are a special case of factory functions. Does your proposed syntax (whatever it is!) treat decorator functions as a special case, or does it apply to general factory functions as well? -- Steven

I am really sorry for splitting the text. My first post was in one piece but got blocked by the moderators. I didn't want to create some ridiculous suspense at the end of the first part. Here is the second part... --- Part 2 - The new-style syntax for decorators: Here is the code of the same decorators as in Part 1 above, but the proposed syntax. The first decorator ('new_style_repeat_fix') is created without parameters, while the second one ('new_style_repeat_var') uses arguments 'n' and 'trace' with the same role as previously: #--- @decorator def new_style_repeat_fix(self, *args, **keys): """docstring for decorating function""" print "apply %r on %r" % (self.deco.__name__, self.func.__name__) for n in range(3): self.func(*args, **keys) @decorator(n=3, trace=True) def new_style_repeat_var(self, *args, **keys): """docstring for decorating function""" if self.trace: print "apply %r on %r" % (self.deco.__name__, self.func.__name__) for n in range(self.n): self.func(*args, **keys) #--- When examining the new-style syntax, one can notice that there are basically four characteristics that distinguish NSD from OSD: * Each NSD is decorated by a newly-introduced callable class called 'decorator' (code provided in Part 3) by using one of two possible decorating forms. The first form '@decorator' is employed for decorators that do not need any parameter. The second form '@decorator(arg1=val1, arg2=val2...)' is employed to specify a sequence of named arguments (combined with their default values) that are passed to the decorator using the standard notation for keyword arguments. * The role of the 'decorator' class is to generate the decorated function (i.e. the inner nested function with the classic OSD syntax) and to broadcast it to the decorating function as its first argument 'self'. When the second decorating form is used, all keyword arguments used in '@decorator(...)' are automatically injected as meta-attributes in this decorated function 'self'. In the example above, the two decorator arguments 'n' and 'trace' are available within the code of 'new_style_repeat_var' as 'self.n' and 'self.trace' respectively. This mechanism avoids one level of nested functions used in standard OSD. * In addition to these decorator arguments, the decorating process also injects two other meta-attributes in the decorated function: 'self.deco' represents a reference to the decorating function, while 'self.func' represents a reference to the undecorated function. If there are several chained decorators, the mechanism is made recursive (more on this later). Note that this implies a slight name restriction: neither 'deco' nor 'func' can be used as the name of a parameter passed to the decorator, as this would generate collision in the corresponding namespace. An alternative might be to consider the two references as "special" attributes and rename them as 'self.__deco__' and 'self.__func__' respectively. I have no clear opinion about the pros/cons of the two alternatives. * Finally, note that each NSD has the same 3-argument signature: (self, *args, **keys). The first argument 'self' has been explained above. The two others 'args' and 'keys' respectively represent the set of positional and keyword arguments, as usual. However, all the values in either 'args' or 'keys' are not meant to be used by the decorating function, but always directly passed to the undecorated function. This means that the statement 'self.func(*args, **keys)' will always appear somewhere in the code of an NSD. Following this mechanism in the decorating function avoids the other level of nested functions used in standard OSD, and guarantees that flat functions are always sufficient. Once the NSD have been defined with the new syntax, they can be used to decorate functions using the standard @-notation, either for single or multiple decoration. For instance: #--- @new_style_repeat_fix def testA(first=0, last=0): """docstring for undecorated function""" print "testA: first=%s last=%s" % (first, last) @new_style_repeat_var(n=5) # 'n' is changed, 'trace' keeps default value def testB(first=0, last=0): """docstring for undecorated function""" print "testB: first=%s last=%s" % (first, last) @new_style_repeat_var # both 'n' and 'trace' keep their default values @new_style_repeat_fix @new_style_repeat_var(n=5, trace=False) # both 'n' and 'trace' are changed def testC(first=0, last=0): """docstring for undecorated function""" print "testC: first=%s last=%s" % (first, last) #--- When applying a decorator without arguments, or when *all* its arguments use their default values, the parenthesis after the decorator name may be dropped. In other words, '@mydeco' and '@mydeco()' are totally equivalent, whether 'mydeco' takes arguments or not. This solves a non-symmetric behavior of standard OSD that has always bothered me: '@old_style_repeat_fix' works but '@old_style_repeat_fix()' does not, and inversely '@old_style_repeat_var()' works but '@old_style_repeat_var' does not. Note also that in the case of chained decorators, each decoration level stores its own set of parameters, so there is no conflict when applying the same decorator several times on the same function, as done with 'new_style_repeat_var' on 'testC'. Now let's play a bit with some introspection tools: #---
testA <function <deco>testA...>
testB <function <deco>testB...>
To explicitely expose the decoration process, a '<deco>' substring is added as a prefix to the '__name__' attribute for each decorated function (more precisely, there is one '<deco>' for each level of decoration, as can be seen with 'testC'). So, each time a '<deco>' prefix is encountered, the user knows that the reference to the corresponding undecorated function (resp. decorating function) is available through the meta-attribute '.func' (resp. '.deco'). When calling 'help' on a decorated function, this principle is clearly displayed, and the user can thus easily obtain useful information, including correct name/signature/docstring: #---
testA.func, testA.deco (<function testA...>, <function new_style_repeat_fix...>)
In the case of chained decorators, the same principle holds recursively. As can be seen in the example below, all information relative to a multi-decorated function (including all decorator arguments used at any decoration level) can be easily fetched by successive applications of the '.func' suffix: #---
testC.func, testC.deco, testC.n, testC.trace (<function <deco><deco>testC...>, <function new_style_repeat_var...>, 3, True)
testC.func.func, testC.func.deco (<function <deco>testC...>, <function new_style_repeat_fix...>)
testC.func.func.func, testC.func.func.deco, testC.func.func.n, testC.func.func.trace (<function testC...>, <function new_style_repeat_var...>, 5, False)
------ to be continued in Part 3... CS

Part 3 - Implementation and additional remarks/questions I've implemented the idea of NSD about nine months ago and have used them successfully in many different situations. My first implementation was terribly messy but at least it did the job. Last week, while thinking about refactoring, I realized that the whole process can be easily implemented as a state machine. This has led me to an extremely compact implementation (about 30 Python statements), where a callable class repeatedly returns reference to itself, until it gets all the required parameters to generate and return the decorated function. Here is the code, so that you can experiment with it, if you wish: #--- class decorator(object): """apply a 'new-style' decorator to a function""" def __init__(self, deco=None, **keys): # either get reference or default parameters for decorating function self.deco = deco; self.keys = keys; self.stack = list() def __call__(self, func=None, **keys): if self.deco is None: # get reference for decorating function self.deco = func; return self elif func is None: # get modified parameters of decorating function self.stack.append(keys); return self else: # get undecorated function and generate decorated function deco = node = lambda *args, **keys: self.deco(deco, *args, **keys) deco.func = func; deco.deco = self.deco; deco.__dict__.update(self.keys) if self.stack: deco.__dict__.update(self.stack.pop()) head = '<deco>'; deco.__name__ = name = head + func.__name__ level = name.count(head); offset = len(head)*level; tail = '.func'*level doc = "use help(%s) to get genuine help" % (name[offset:] + tail) while hasattr(node, 'func'): node.__doc__ = doc; node = node.func return deco #--- The simplicity of the new implementation has convinced me that it might be useful to share this idea and write a proposal in order to get some feedback from the community. As said in the introduction, this is my first post to python-ideas, so I'm not sure about the correct process to follow. I've got plenty of questions anyway: * Is the idea interesting enough to deserve consideration for possible inclusion in the language? If yes, should I transform this proposal into a PEP, or should there first be some pre-PEP discussion here (or maybe in python-dev)? * Are there some pitfalls involved with the use of NSD that I haven't seen? Or are there additional desirable elements that could be easily included? * After having read this proposal, has anybody some suggestion for alternative syntax that offer similar features? * There are some advanced features offered by the new syntax (such as meta-decorator, or whatever you call them), which seem to be powerful but are less stable than the elements presented here. I did not detail this kind of stuff because I consider that it is likely to create some noise in the discussion, but maybe not? Thanks for any opinion, CS

Christophe Schlick wrote:
Decorators were one of the biggest successes in recent years, so we would be foolish to dismiss the idea of simplifying them out of hand. But I think you are doing yourself a disservice by referring to this proposal as "new syntax". Normally when people talk about syntax, they are referring to language syntax (i.e. a change to the Python interpreter), and we're pretty conservative about adding new syntax. It seems to me that you're talking about a new idiom for building decorator functions, not new syntax. I would suggest you also publish this decorator-builder recipe on ActiveState's Python cookbook, and see if you get much interest there. It might also help to post a link to your recipe to python-list@python.org. You certainly should do those things before going to python-dev.
Have you timed the decorated function using new and old style? If you decorator a function with (say) 5 arguments, is there any performance hit to your NSD? Do you have any tests for it? E.g. unit tests, regression tests? Your code looks opaque and complicated to me, I would want to see a good test suite before even considering using it in production code, let alone in the standard library. -- Steven

On Tue, Apr 26, 2011 at 8:08 PM, Steven D'Aprano <steve@pearwood.info> wrote:
You are right. I've mostly used the expression "new-style decorators" (which corresponds to the idea of "new idiom" that you propose) but I guess that there are still a couple of unwanted "new syntax" in the post. At the beginning of the post, I say that "no existing syntax is broken", so the proposal does not change anything for the interpreter. Moreover, the old idiom can be used in combination with the new one, so no existing code can be broken by the proposal. CS

On Tue, Apr 26, 2011 at 8:08 PM, Steven D'Aprano <steve@pearwood.info> wrote:
OK, I'm going to try that. Thanks.
Intuitively I would say the the only performance hit would comme from the fact that the decorator arguments are accessed via self.__dict__ in NSD, while there are available as locals with OSD. I've made some quick 'timeit' tests. I don't know if this is the kind of timing you thought about: #--- from timeit import Timer from decorator import decorator # OSD def old_add_args(a=1, b=2, c=3, d=4, e=5): def dummy1(func): def dummy2(*args, **keys): return a + b + c + d + e + func(*args, **keys) return dummy2 return dummy1 # NSD @decorator(a=1, b=2, c=3, d=4, e=5) def new_add_args(self, *args, **keys): return self.a + self.b + self.c + self.d + self.e + self.func(*args, **keys) # Apply OSD @old_add_args() def old_test(*args, **keys): return sum(*args) # Apply NSD @new_add_args() def new_test(*args, **keys): return sum(*args) # Gentle case: the evaluation of the function is rather long compared to the time # needed to fetch the 5 decorator args old_time = Timer('old_test(range(999))', 'from __main__ import old_test').timeit() new_time = Timer('new_test(range(999))', 'from __main__ import new_test').timeit() print "Gentle: old = %.3f new = %.3f" % (old_time, new_time) # Worst case: the evaluation of the function is negligible compared to the time # needed to get the 5 decorators args. old_time = Timer('old_test(range(1))', 'from __main__ import old_test').timeit() new_time = Timer('new_test(range(1))', 'from __main__ import new_test').timeit() print "Worst: old = %.3f new = %.3f" % (old_time, new_time) #--- Here are the timings obtained on my notebook: Gentle: old = 45.983 new = 46.377 Worst: old = 4.043 new = 5.127 which seems to confirm that the overhead mainly comes from the 'self.xxx' fetch, and it pretty negligible when heavy computation is performed. CS

Just before I go to bed, here some alternative implementations: Usage: ------ # for argument-less decorators: @decorator def my_deco(func,*func_args,**func_kwargs): pass # of course regular arguments can also be declared (same for all funcs below) @my_deco def func(*func_args,**func_kwargs): pass # for decorators with arguments: # even when there are no default arguments function-call parenthesis are needed @decorator_with_args('foo',bar='baz') def my_deco(func,deco_args,deco_kwargs,*func_args,**func_kwargs): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass # alternative version where the decorator arguments are expanded: # `...` is a placeholder for the arguments of the decorator in regular argument syntax. # This way the decorator arguments can be declared inline and no deco_(kw)args or self.* # is needed. Also decorator arguments are not decoupled from their default values this way. @decorator_with_expanded_args def my_deco(func,...,*func_args,**func_kwargs): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass Implementation: --------------- from types import FunctionType, ClassType from functools import wraps def decorator(deco): @wraps(deco) def _deco(func): @wraps(func) def _f(*args,**kwargs): return deco(func,*args,**kwargs) return _f return _deco def decorator_with_args(*deco_default_args,**deco_default_kwargs): def _deco_deco_deco(deco): @wraps(deco) def _deco_deco(*deco_args,**deco_kwargs): if len(deco_args) < len(deco_default_args): deco_args = deco_args+deco_default_args[len(deco_args):] merged_deco_kwargs = dict(deco_default_kwargs) merged_deco_kwargs.update(deco_kwargs) del deco_kwargs def _deco(func): @wraps(func) def _f(*args,**kwargs): return deco( func,deco_args,merged_deco_kwargs,*args,**kwargs) return _f return _deco return _deco_deco return _deco_deco_deco def decorator_with_expanded_args(deco): if isinstance(deco, FunctionType): co = deco.func_code deco_name = deco.func_name arg_names = list(co.co_varnames[0:co.co_argcount]) elif isinstance(deco, ClassType): co = deco.__init__.func_code deco_name = deco.__name__ arg_names = list(co.co_varnames[1:co.co_argcount]) elif hasattr(deco, '__call__'): co = deco.__call__.func_code deco_name = type(deco).__name__ arg_names = list(co.co_varnames[0:co.co_argcount]) else: raise TypeError('not a decorateable object') if not arg_names: raise TypeError('decorator function needs a func argument') del co del arg_names[0] min_argcount = len(arg_names) if deco.func_defaults: min_argcount -= len(deco.func_defaults) @wraps(deco) def _deco_deco(*args,**kwargs): deco_args = list(args) n = len(deco_args) if n < len(arg_names): i = n - min_argcount for arg in arg_names[n:]: if arg in kwargs: deco_args.append(kwargs.pop(arg)) elif i < 0: raise TypeError( '%s() takes at least %d positional ' + 'arguments (%d given)' % (deco_name, min_argcount, len(deco_args))) else: deco_args.append(deco.func_defaults[i]) i += 1 if kwargs: arg = kwargs.keys()[0] if arg in arg_names: raise TypeError( "%s() got multiple values for keyword argument '%s'" % (deco_name, arg)) else: raise TypeError("%s() got an unexpected keyword argument '%s'" % (deco_name, arg)) deco_args = tuple(deco_args) def _deco(func): @wraps(func) def _f(*args,**kwargs): return deco(func,*(deco_args+args),**kwargs) return _f return _deco return _deco_deco What do you think? -panzi

On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick@gmail.com>wrote:
I'm sure it's not exactly as you envisioned, but the decorator package provides roughly what you're describing: http://micheles.googlecode.com/hg/decorator/documentation.html#decorator-is-... -- Daniel Stutzbach

On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick@gmail.com> wrote: <snip>
I'm personally not a fan of magic such as having both @decorator and @decorator(...) work.
What if my decorator's parameters don't all have default values? What if I don't want people to have to use keyword arguments when using my decorator? What if my decorator accepts extra positional arguments (i.e. *args)? Cheers, Chris

On Wed, Apr 27, 2011 at 12:30 AM, Chris Rebert <pyideas@rebertia.com> wrote:
Well, you may simply use @decorator(), if your decorator does not need arguments. Actually the magic does not come from the new 'decorator' class, but directly from the state machine used to parse the @-syntax in Python. I am not responsible for that.
The idea behind the proposal is to reduce the boilerplate for most of the standard patterns of decorators. One element of that reduction is to automatically transform the decorator arguments as attributes of the decorated function. To do this, the attributes have to get individual names, that's why I've proposed the keyword argument syntax. However, it is actually possible to implement the same idea by letting the decorator use positional arguments, which are then combined into a single tuple attribute 'self.args' available for the decorating function. The code of the 'decorator' class would simply be a bit longer, but there is no specific difficulty here. CS

On 4/26/2011 11:05 AM, Christophe Schlick wrote: I got interrupted in responding this, and you have now posted parts 2 and 3, so I will snip and revise a bit.
a new syntax for defining decorators in Python.
There is no special syntax for defining decorators -- just normal nested function or class definition syntax. To put it another way, Python does not have decorator objects. A decorator is simply a callable (function, class, or class instance with __call__ method) applied to a object (function or class) with the @deco syntax (before the def/class statement) instead of the normal call syntax (after the def/class statement). Decorators return either the original object (usually modified) or another object that is usually, but not necessarily, of the same kind as the input. As Stephen noted, syntax is what is defined in the Language reference. Code patterns are what are embodied in the stdlib (or pypi or the Python cookbook or other example repositories). What you are actually proposing is a meta-decorator (class) whose instances can be used as decorators because the class has a __call__ instance method. This sort of thing is a known alternative to the nested function pattern.
programmer has no other choice than to create a dummy function name
Many of us consider dummy names a bogus problem. I recommend you skip this argument. In any case, this, like the other 'problems' you describe for nested functions, has nothing in particular with their usage as decorators.
(only used for one single 'return' statement), which is never a good coding principle, whatever the programming language.
This strikes me as a bogus rule: a single use of a local name is quite common, and not just in Python. I recommend leaving this also out of your problem list. Stick with the two real problems. 1. The double or triple nested function pattern has a lot of boilerplate and can be difficult to learn. Hiding boilerplate in a class makes the use pattern simpler and easier to learn. This is a real benefit. One of the three major benefits of writing a generator function versus an equivalent iterator class is that is hides the boilerplate required for the latter. Similarly, for loops hide the boilerplate required for an equivalent while loop. 2. Introspection (more comments below), which your class also addresses.
Actually, this is the docstring for the decorator making function.
This time a two-level function nesting is required and the code needs two dummy names for these two nested functions.
'deco' and 'wrapper' work for me. But I agree that this is a bit confusing. But I think that is somewhat inherent in calling a decorator-maker f1 to return decorator f2 that returns wrapper f3 that wraps the original function f.
Note that the docstring of the middle nested function is even totally invisible for introspection tools.
Not true. To see the docstring of a dynamically created temporary function, you have to either dynamically create it or dig inside the function that creates it to find the constant string:
old_style_repeat_var().__doc__ 'docstring never used'
old_style_repeat_var.__code__.co_consts[1].co_consts[0] 'docstring never used'
But I am not sure why you would want to see it or even have one.
The 'problems' of nested defs has nothing to do with decorators in particular. Functional programmers use them all the time.
Only because you commented out @wraps. Again, this is not a problem of the @decorator syntax but of *all* wrapping callables. Functools.partial has the same 'problem'.
I am not sure what you mean. If the two signatures are different, then one must use the signature of the wrapper when calling it, not the signature of the wrappee, which is perhaps what you mean by 'the genuine one'. The problem of generic wrappers having generic signatures (*args, **kwds) is endemic to using generic wrappers instead of special case wrappers.
reason is that signature copying is not an easy process.
True if you want to do it generically. Copying with modification, as functools.partial would have to do, is even worse.
The other solution is to not use a generic wrappers with generic signatures but to write specific wrappers with the actual signature, which people did, for instance, before functools and partial() were added to Python. There have been proposals but no consensus on a decorator or decolib module for the stdlib. I second the other recommendations to make your proposal available on the cookbook site, etc. -- Terry Jan Reedy

I thought about this problem for I wile and I came up with this @decorator: Usage: ------
Or a more specific example:
This fully supports *args and **kwargs besides regular arguments for the decorator and the decorated function. By that I mean func_args already contains the filled in default values of func as well as any regular arguments that where passed as keyword argument and argument passing errors are handled (e.g. passing an argument as positional and keyword argument). Error handling example:
Traceback (most recent call last): File "<pyshell#114>", line 1, in <module> @deco(1,2,3) File "<pyshell#96>", line 22, in _deco_deco deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs) TypeError: deco() takes at most 2 arguments (3 given) Or:
foo(5,6,y=33)
Traceback (most recent call last): File "<pyshell#112>", line 1, in <module> foo(5,6,y=33) File "<pyshell#96>", line 27, in _f func_args, func_kwargs = apply_func_args(*func_args, **func_kwargs) TypeError: foo() got multiple values for keyword argument 'y' Of course that always needs function call parenthesis on the decorator, even if the decorator does not take any arguments. Maybe it could be extended that in this case a more simple decorator mechanism is used. A decorator-decorator for decorators without arguments would be very simple (see end of mail). Implementation: --------------- from types import FunctionType, ClassType from functools import wraps def inspect_callable(func): """-> (arg_names, co_flags, func_defaults, func_name)""" return _inspect_callable(func,set()) def _inspect_callable(func,visited): if func in visited: raise TypeError("'%s' object is not callable" % type(func).__name__) visited.add(func) if isinstance(func, FunctionType): co = func.func_code func_name = func.__name__ arg_names = list(co.co_varnames[0:co.co_argcount]) defaults = func.func_defaults flags = co.co_flags elif isinstance(func, ClassType): func_name = func.__name__ arg_names, flags, defaults, member_name = _inspect_callable(func.__init__,visited) if arg_names: del arg_names[0] elif hasattr(func, '__call__'): func_name = '<%s object at 0x%x>' % (type(func).__name__, id(func)) arg_names, flags, defaults, member_name = _inspect_callable(func.__call__,visited) else: raise TypeError("'%s' object is not callable" % type(func).__name__) return arg_names, flags, defaults, func_name FUNC_ARGS = 0x04 FUNC_KWARGS = 0x08 FUNC_GEN = 0x20 # this function should probably be reimplemented in C: def args_applyer(arg_names,flags=0,defaults=None,func_name=None): """-> f(args..., [*varargs], [**kwargs]) -> ((args...)+varargs, kwargs)""" all_args = list(arg_names) if arg_names: body = ['(',','.join(arg_names),')'] else: body = [] if flags & FUNC_ARGS: args_name = '_args' i = 0 while args_name in arg_names: args_name = '_args'+i i += 1 all_args.append('*'+args_name) if arg_names: body.append('+') body.append(args_name) elif not arg_names: body.append('()') body.append(',') if flags & FUNC_KWARGS: kwargs_name = '_kwargs' i = 0 while kwargs_name in arg_names: kwargs_name = '_kwargs'+i i += 1 all_args.append('**'+kwargs_name) body.append(kwargs_name) else: body.append('{}') if func_name: apply_args = named_lambda(func_name,all_args,''.join(body)) else: apply_args = eval('lambda %s: (%s)' % (','.join(all_args), ''.join(body))) if defaults: apply_args.func_defaults = defaults return apply_args def named_lambda(name,args,body): code = 'def _named_lambda():\n\tdef %s(%s):\n\t\treturn %s\n\treturn %s' % ( name, ','.join(args), body, name) del name, args, body exec(code) return _named_lambda() # begin helper functions (not used by this module but might be handy for decorator developers) def args_applyer_for(func): return args_applyer(*inspect_callable(func)) def apply_args(args,kwargs,arg_names,flags=0,defaults=None,func_name=None): return args_applyer(arg_names,flags,defaults,func_name)(*args,**kwargs) def apply_args_for(func,args,kwargs): return args_applyer(*inspect_callable(func))(*args,**kwargs) # end helper functions def decorator(deco): """deco(func,func_args,func_kwargs,deco_args...) @decorator def my_deco(func,func_args,func_kwargs,deco_args...): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass """ arg_names, flags, defaults, deco_name = inspect_callable(deco) if flags & FUNC_ARGS == 0: if len(arg_names) < 3: raise TypeError('decorator functions need at least 3 ' + 'arguments (func, func_args, func_kwargs)') del arg_names[0:3] apply_deco_args = args_applyer(arg_names,flags,defaults,deco_name) del flags, defaults @wraps(deco) def _deco_deco(*deco_args,**deco_kwargs): deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*func_args,**func_kwargs): func_args, func_kwargs = apply_func_args(*func_args, **func_kwargs) return deco(func,func_args,func_kwargs,*deco_args,**deco_kwargs) return _f return _deco return _deco_deco def simple_decorator(deco): """deco(func,func_args,func_kwargs) @simple_decorator def my_deco(func,func_args,func_kwargs): pass @my_deco def func(*func_args,**func_kwargs): pass """ @wraps(deco) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*args,**kwargs): return deco(func,*apply_func_args(*args,**kwargs)) return _f return _deco

Maybe even better, decorator and simple_decorator in one: def decorator(deco): """deco(func,func_args,func_kwargs,deco_args...) @decorator def my_deco(func,func_args,func_kwargs,deco_args...): pass @my_deco(*deco_args,**deco_kwargs) def func(*func_args,**func_kwargs): pass @decorator def my_deco2(func,func_args,func_kwargs): pass @my_deco2 def func2(*func_args,**func_kwargs): pass @decorator def my_deco3(func,func_args,func_kwargs,x=1): pass @my_deco3() def func3(*func_args,**func_kwargs): pass """ arg_names, flags, defaults, deco_name = inspect_callable(deco) if flags & FUNC_ARGS == 0: if len(arg_names) < 3: raise TypeError('decorator functions need at least 3 ' + 'arguments (func, func_args, func_kwargs)') del arg_names[0:3] if not arg_names and flags & (FUNC_ARGS | FUNC_KWARGS) == 0: # argument-less decorator del arg_names, flags, defaults, deco_name @wraps(deco) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*args,**kwargs): return deco(func,*apply_func_args(*args,**kwargs)) return _f return _deco else: apply_deco_args = args_applyer(arg_names,flags,defaults,deco_name) del arg_names, flags, defaults, deco_name @wraps(deco) def _deco_deco(*deco_args,**deco_kwargs): deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs) def _deco(func): apply_func_args = args_applyer(*inspect_callable(func)) @wraps(func) def _f(*func_args,**func_kwargs): func_args, func_kwargs = apply_func_args( *func_args, **func_kwargs) return deco(func,func_args,func_kwargs, *deco_args,**deco_kwargs) return _f return _deco return _deco_deco
participants (7)
-
Chris Rebert
-
Christophe Schlick
-
Daniel Stutzbach
-
Ethan Furman
-
Mathias Panzenböck
-
Steven D'Aprano
-
Terry Reedy