[Python-Dev] (my) revisions to PEP318 finally done.
Anthony Baxter
anthony at interlink.com.au
Mon Aug 30 15:22:10 CEST 2004
Here's the version of PEP 318 I just checked in. If you have any
additions or changes, I prefer patches mailed to me in unified
diff format. Comments like "You should cover point X, Y and Z",
without the accompanying words, I won't be acting on (others with
checkin privs are more than welcome to do so). Sorry about that,
but I _cannot_ spend any more time on this document now - this
whole decorators thing has consumed far more of my time than I
can afford right now.
-------------- next part --------------
PEP: 318
Title: Decorators for Functions and Methods
Version: $Revision: 1.28 $
Last-Modified: $Date: 2004/08/30 13:16:56 $
Author: Kevin D. Smith, Jim Jewett, Skip Montanaro, Anthony Baxter
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 05-Jun-2003
Python-Version: 2.4
Post-History: 09-Jun-2003, 10-Jun-2003, 27-Feb-2004, 23-Mar-2004, 30-Aug-2004
WarningWarningWarning
=====================
The final decision on the syntax for 2.4a3 is not yet made. This
will be done before 2.4a3, and this document will be updated to
match. Note also that this document does not attempt to cover the
huge number of potential alternative syntaxes, nor is it an attempt
to exhaustively list all the positives and negatives of each form.
Abstract
========
The current method for transforming functions and methods (for
instance, declaring them as a class or static method) is awkward and
can lead to code that is difficult to understand. Ideally, these
transformations should be made at the same point in the code where the
declaration itself is made. This PEP introduces new syntax for
transformations of a function or method declaration.
Motivation
==========
The current method of applying a transformation to a function or
method places the actual translation after the function body. For
large functions this separates a key component of the function's
behavior from the definition of the rest of the function's external
interface. For example::
def foo(self):
perform method operation
foo = classmethod(foo)
This becomes less readable with longer methods. It also seems less
than pythonic to name the function three times for what is
conceptually a single declaration. A solution to this problem is to
move the transformation of the method closer to the method's own
declaration. While the new syntax is not yet final, the intent is to
replace::
def foo(cls):
pass
foo = synchronized(lock)(foo)
foo = classmethod(foo)
with an alternative that places the decoration in the function's
declaration::
@classmethod
@synchronized(lock)
def foo(cls):
pass
Modifying classes in this fashion is also possible, though the
benefits are not as immediately apparent. Almost certainly, anything
which could be done with class decorators could be done using
metaclasses, but using metaclasses is sufficiently obscure that there
is some attraction to having an easier way to make simple
modifications to classes. For Python 2.4, only function/method
decorators are being added.
Why Is This So Hard?
--------------------
Two decorators (``classmethod()`` and ``staticmethod()``) have
been available in Python since version 2.2. It's been assumed since
approximately that time that some syntactic support for them would
eventually be added to the language. Given this assumption, one might
wonder why it's been so difficult to arrive at a consensus.
Discussions have raged off-and-on at times in both comp.lang.python
and the python-dev mailing list about how best to implement function
decorators. There is no one clear reason why this should be so, but a
few problems seem to be most problematic.
* Disagreement about where the "declaration of intent" belongs.
Almost everyone agrees that decorating/transforming a function at
the end of its definition is suboptimal. Beyond that there seems to
be no clear consensus where to place this information.
* Syntactic constraints. Python is a syntactically simple language
with fairly strong constraints on what can and can't be done without
"messing things up" (both visually and with regards to the language
parser). There's no obvious way to structure this information so
that people new to the concept will think, "Oh yeah, I know what
you're doing." The best that seems possible is to keep new users
from creating a wildly incorrect mental model of what the syntax
means.
* Overall unfamiliarity with the concept. For people who have a
passing acquaintance with algebra (or even basic arithmetic) or have
used at least one other programming language, much of Python is
intuitive. Very few people will have had any experience with the
decorator concept before encountering it in Python. There's just no
strong preexisting meme that captures the concept.
* Syntax discussions in general appear to cause more contention than
almost anything else. Readers are pointed to the ternary operator
discussions that were associated with PEP 308 for another example of
this.
Background
==========
There is general agreement that syntactic support is desirable to the
current state of affairs. Guido mentioned `syntactic support for
decorators`_ in his DevDay keynote presentation at the `10th Python
Conference`_, though `he later said`_ it was only one of several
extensions he proposed there "semi-jokingly". `Michael Hudson raised
the topic`_ on ``python-dev`` shortly after the conference,
attributing the initial bracketed syntax to an earlier proposal on
``comp.lang.python`` by `Gareth McCaughan`_.
.. _syntactic support for decorators:
http://www.python.org/doc/essays/ppt/python10/py10keynote.pdf
.. _10th python conference:
http://www.python.org/workshops/2002-02/
.. _michael hudson raised the topic:
http://mail.python.org/pipermail/python-dev/2002-February/020005.html
.. _he later said:
http://mail.python.org/pipermail/python-dev/2002-February/020017.html
.. _gareth mccaughan:
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=slrna40k88.2h9o.Gareth.McCaughan%40g.local
Class decorations seem like an obvious next step because class
definition and function definition are syntactically similar.
The discussion continued on and off on python-dev from February 2002
through July 2004. Hundreds and hundreds of posts were made, with
people proposing many possible syntax variations. Guido took a list
of proposals to `EuroPython 2004`_, where a discussion took place.
Subsequent to this, he decided that for 2.4a2 we'd have the
`Java-style`_ @decorator syntax. Barry Warsaw named this the
'pie-decorator' syntax, in honor of the Pie-thon Parrot shootout which
was announced about the same time as the decorator syntax, and because
the @ looks a little like a pie. Guido `outlined his case`_ on
Python-dev, including `this piece`_ on the various rejected forms.
.. _EuroPython 2004:
http://www.python.org/doc/essays/ppt/euro2004/euro2004.pdf
.. _outlined his case:
http://mail.python.org/pipermail/python-dev/2004-August/author.html
.. _this piece:
http://mail.python.org/pipermail/python-dev/2004-August/046672.html
.. _Java-style:
http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html
On the name 'Decorator'
=======================
There's been a number of complaints about the choice of the name
'decorator' for this feature. The major one is that the name is not
consistent with its use in the `GoF book`_. The name 'decorator'
probably owes more to its use in the compiler area -- a syntax tree is
walked and annotated. It's quite possible that a better name may turn
up.
.. _GoF book:
http://patterndigest.com/patterns/Decorator.html
Design Goals
============
The new syntax should
* work for arbitrary wrappers, including user-defined callables and
the existing builtins ``classmethod()`` and ``staticmethod()``. This
requirement also means that a decorator syntax must support passing
arguments to the wrapper constructor
* work with multiple wrappers per definition
* make it obvious what is happening; at the very least it should be
obvious that new users can safely ignore it when writing their own
code
* be a syntax "that ... [is] easy to remember once explained"
* not make future extensions more difficult
* be easy to type; programs that use it are expected to use it very
frequently
* not make it more difficult to scan through code quickly. It should
still be easy to search for all definitions, a particular
definition, or the arguments that a function accepts
* not needlessly complicate secondary support tools such as
language-sensitive editors and other "`toy parser tools out
there`_"
* allow future compilers to optimize for decorators. With the hope of
a JIT compiler for Python coming into existence at some point this
tends to require the syntax for decorators to come before the
function definition
* move from the end of the function, where it's currently hidden, to
the front where it is more `in your face`_
Andrew Kuchling has links to a bunch of the discussions about
motivations and use cases `in his blog`_. Particularly notable is `Jim
Huginin's list of use cases`_.
.. _toy parser tools out there:
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=mailman.1010809396.32158.python-list%40python.org
.. _in your face:
http://mail.python.org/pipermail/python-dev/2004-August/047112.html
.. _in his blog:
http://www.amk.ca/diary/archives/cat_python.html#003255
.. _Jim Huginin's list of use cases:
http://mail.python.org/pipermail/python-dev/2004-April/044132.html
Current Syntax
==============
The current syntax for function decorators as implemented in Python
2.4a2 is::
@dec2
@dec1
def func(arg1, arg2, ...):
pass
This is equivalent to::
def func(arg1, arg2, ...):
pass
func = dec2(dec1(func))
without the intermediate assignment to the variable ``func``. The
decorators are near the function declaration. The @ sign makes it
clear that something new is going on here.
The decorator statement is limited in what it can accept -- arbitrary
expressions will not work. Guido preferred this because of a `gut
feeling`_.
.. _gut feeling:
http://mail.python.org/pipermail/python-dev/2004-August/046711.html
Syntax Alternatives
===================
There have been `a large number`_ of different syntaxes proposed --
rather than attempting to work through these individual syntaxes, it's
worthwhile to break the syntax discussion down into a number of areas.
Attempting to discuss `each possible syntax`_ individually would be an
act of madness, and produce a completely unwieldy PEP.
.. _a large number:
http://www.python.org/moin/PythonDecorators
.. _each possible syntax:
http://ucsu.colorado.edu/~bethard/py/decorators-output.py
Decorator Location
------------------
The first syntax point is the location of the decorators. For the
following examples, we use the @syntax used in 2.4a2.
Decorators before the def statement are the first alternative,
and the syntax used in 2.4a2::
@classmethod
def foo(arg1,arg2):
pass
@accepts(int,int)
@returns(float)
def bar(low,high):
pass
There have been a number of objections raised to this location --
the primary one is that it's the first real Python case where a
line of code has a result on a following line. The syntax that
will be in 2.4a3 will also require one decorator per line (in a2,
multiple decorators can be specified on the same line).
People also complained that the syntax got unworldly quickly when
multiple decorators were used. The point was made, though, that the
chances of a large number of decorators being used on a single function
were small and thus this was not a large worry.
Some of the advantages of this form are that the decorators live
outside the method body -- they are obviously executed at the time
the function is defined.
Another advantage is that being prefix to the function definition fit the
idea of knowing about a change to the semantics of the code before the
code itself, thus knowing how to interpret the code's semantics
properly without having to go back and change your initial perceptions
if the syntax did not come before the function definition.
Guido decided `he preferred`_ having the decorators on the line before
the 'def', because it was felt that a long argument list would mean
that the decorators would be 'hidden'
.. _he preferred:
http://mail.python.org/pipermail/python-dev/2004-March/043756.html
The second form is the decorators between the def and the function
name, or the function name and the argument list::
def @classmethod foo(arg1,arg2):
pass
def @accepts(int,int), at returns(float) bar(low,high):
pass
def foo @classmethod (arg1,arg2):
pass
def bar @accepts(int,int), at returns(float) (low,high):
pass
There are a couple of objections to this form. The first is that it
breaks easily 'greppability' of the source -- you can no longer search
for 'def foo(' and find the definition of the function. The second,
more serious, objection is that in the case of multiple decorators,
the syntax would be extremely unwieldy.
The next form, which has had a number of strong proponents, is to
have the decorators between the argument list and the trailing ``:``
in the 'def' line::
def foo(arg1,arg2) @classmethod:
pass
def bar(low,high) @accepts(int,int), at returns(float):
pass
Guido `summarized the arguments`_ against this form (many of which
also apply to the previous form) as:
- it hides crucial information (e.g. that it is a static method)
after the signature, where it is easily missed
- it's easy to miss the transition between a long argument list and a
long decorator list
- it's cumbersome to cut and paste a decorator list for reuse, because
it starts and ends in the middle of a line
.. _summarized the arguments:
http://mail.python.org/pipermail/python-dev/2004-August/047112.html
The next form is that the decorator syntax go inside the method
body at the start, in the same place that docstrings currently
live:
def foo(arg1,arg2):
@classmethod
pass
def bar(low,high):
@accepts(int,int)
@returns(float)
pass
The primary objection to this form is that it requires "peeking
inside" the method body to determine the decorators. In addition,
even though the code is inside the method body, it is not executed
when the method is run. Guido felt that docstrings were not a good
counter-example, and that it was quite possible that a 'docstring'
decorator could help move the docstring to outside the function body.
The final form is a new block that encloses the method's code. For
this example, we'll use a 'decorate' keyword, as it makes no sense
with the @syntax. ::
decorate:
classmethod
def foo(arg1,arg2):
pass
decorate:
accepts(int,int)
returns(float)
def bar(low,high):
pass
This form would result in inconsistent indentation for decorated and
undecorated methods. In addition, a decorated method's body would
start three indent levels in.
Syntax forms
------------
* ``@decorator``::
@classmethod
def foo(arg1,arg2):
pass
@accepts(int,int)
@returns(float)
def bar(low,high):
pass
The major objections against this syntax are that the @ symbol is
not currently used in Python (and is used in both IPython and Leo),
and that the @ symbol is not meaningful. Another objection is that
this "wastes" a currently unused character (from a limited set) on
something that is not perceived as a major use.
* ``|decorator``::
|classmethod
def foo(arg1,arg2):
pass
|accepts(int,int)
|returns(float)
def bar(low,high):
pass
This is a variant on the @decorator syntax -- it has the advantage
that it does not break IPython and Leo. Its major disadvantage
compared to the @syntax is that the | symbol looks like both a
capital I and a lowercase l.
* list syntax::
[classmethod]
def foo(arg1,arg2):
pass
[accepts(int,int), returns(float)]
def bar(low,high):
pass
The major objection to the list syntax is that it's currently
meaningful (when used in the form before the method). It's also
lacking any indication that the expression is a decorator.
* list syntax using other brackets (``<...>``, ``[[...]]``, ...)::
<classmethod>
def foo(arg1,arg2):
pass
<accepts(int,int), returns(float)>
def bar(low,high):
pass
None of these alternatives gained much traction. The alternatives
which involve square brackets only serve to make it obvious that the
decorator construct is not a list. They do nothing to make parsing any
easier. The '<...>' alternative presents parsing problems because '<'
and '>' already parse as un-paired. They present a further parsing
ambiguity because a right angle bracket might be a greater than symbol
instead of a closer for the decorators.
* ``decorate()``
The ``decorate()`` proposal was that no new syntax be implemented --
instead a magic function that used introspection to manipulate the
following function. Both Jp Calderone and Philip Eby produced
implementations of functions that did this. Guido was pretty firmly
against this -- with no new syntax, the magicness of a function like
this is extremely high:
Using functions with "action-at-a-distance" through
sys.settraceback may be okay for an obscure feature that can't be
had any other way yet doesn't merit changes to the language, but
that's not the situation for decorators. The widely held view
here is that decorators need to be added as a syntactic feature to
avoid the problems with the postfix notation used in 2.2 and 2.3.
Decorators are slated to be an important new language feature and
their design needs to be forward-looking, not constrained by what
can be implemented in 2.3.
* new keyword (and block)
This idea was the consensus alternate from comp.lang.python. Robert
Brewer wrote up a detailed `J2 proposal`_ document outlining the arguments
in favor of this. The issues with this form are that it requires a new
keyword (and therefore a ``from __future__ import decorators`` statement),
it seems like there is no obvious choice for the keyword (``using`` is
the choice of the proposal and implementation), and that the form
produces something that looks like a normal code block, but isn't. Attempts
to use statements in this block will cause a syntax error. It's also
going to potentially confuse users who don't expect a block to contain
bare expressions.
.. _J2 proposal:
http://www.aminus.org/rbre/python/pydec.html
There are plenty of other variants and proposals on `the wiki page`_.
.. _the wiki page:
http://www.python.org/moin/PythonDecorators
Why @?
------
There is some history in Java using @ initially as a marker in
`Javadoc comments`_ and later in Java 1.5 for `annotations`_, which
are similar to Python decorators. The fact that @ was previously
unused as a token in Python also means it's clear there is no
possibility of such code being parsed by an earlier version of Python,
leading to possibly subtle semantic bugs. It also means that ambiguity
of what is a decorator and what isn't is removed. of That said, @ is
still a fairly arbitrary choice. Some have suggested using | instead.
For syntax options which use a list-like syntax (no matter where it
appears) to specify the decorators a few alternatives were proposed:
``[|...|]``, ``*[...]*``, and ``<...>``.
.. _Javadoc comments:
http://java.sun.com/j2se/javadoc/writingdoccomments/
.. _annotations:
http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html
Current Implementation, History
===============================
Guido asked for a volunteer to implement his preferred syntax, and Mark
Russell stepped up and posted a `patch`_ to SF. The syntax accepted
for 2.4a2 is::
@dec2
@dec1
def func(arg1, arg2, ...):
pass
This is equivalent to::
def func(arg1, arg2, ...):
pass
func = dec2(dec1(func))
though without the intermediate creation of a variable named ``func``.
A `previous patch`_ from Michael Hudson which implements the
list-after-def syntax is also still kicking around.
.. _patch: http://www.python.org/sf/979728
.. _previous patch: http://starship.python.net/crew/mwh/hacks/meth-syntax-sugar-3.diff
After 2.4a2 was released, in response to community reaction, Guido
stated that he'd re-examine a community proposal, if the community
could come up with a community consensus, a decent proposal, and an
implementation.
After an amazing number of posts, collecting a vast number of alternatives
in the `Python wiki`_, the proposed J2 syntax (the new keyword ``using``
in a block before the def). Robert Brewer wrote a `detailed proposal`_
for this form, and Michael Sparks produced `a patch`_. As at time of
writing, we're waiting for Guido's decision.
.. _Python wiki:
http://www.python.org/moin/PythonDecorators
.. _detailed proposal:
http://www.aminus.org/rbre/python/pydec.html
.. _a patch:
http://www.python.org/sf/1013835
Examples
========
Much of the discussion on ``comp.lang.python`` and the ``python-dev``
mailing list focuses on the use of decorators as a cleaner way to use
the ``staticmethod()`` and ``classmethod()`` builtins. This
capability is much more powerful than that. This section presents
some examples of use.
1. Define a function to be executed at exit. Note that the function
isn't actually "wrapped" in the usual sense. ::
def onexit(f):
import atexit
atexit.register(f)
return f
@onexit
def func():
...
Note that this example is probably not suitable for real usage, but
is for example purposes only.
2. Define a class with a singleton instance. Note that once the class
disappears enterprising programmers would have to be more creative
to create more instances. (From Shane Hathaway on ``python-dev``.)
::
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
@singleton
class MyClass:
...
3. Add attributes to a function. (Based on an example posted by
Anders Munch on ``python-dev``.) ::
def attrs(**kwds):
def decorate(f):
for k in kwds:
setattr(f, k, kwds[k])
return f
return decorate
@attrs(versionadded="2.2",
author="Guido van Rossum")
def mymethod(f):
...
4. Enforce function argument and return types. Note that this
copies the func_name attribute from the old to the new function.
func_name was made writable in Python 2.4a3::
def accepts(*types):
def check_accepts(f):
assert len(types) == f.func_code.co_argcount
def new_f(*args, **kwds):
for (a, t) in zip(args, types):
assert isinstance(a, t), \
"arg %r does not match %s" % (a,t)
return f(*args, **kwds)
new_f.func_name = f.func_name
return new_f
return check_accepts
def returns(rtype):
def check_returns(f):
def new_f(*args, **kwds):
result = f(*args, **kwds)
assert isinstance(result, rtype), \
"return value %r does not match %s" % (result,rtype)
return result
new_f.func_name = f.func_name
return new_f
return check_returns
@accepts(int, (int,float))
@returns((int,float))
def func(arg1, arg2):
return arg1 * arg2
5. Declare that a class implements a particular (set of) interface(s).
This is from a posting by Bob Ippolito on ``python-dev`` based on
experience with `PyProtocols`_. ::
def provides(*interfaces):
"""
An actual, working, implementation of provides for
the current implementation of PyProtocols. Not
particularly important for the PEP text.
"""
def provides(typ):
declareImplementation(typ, instancesProvide=interfaces)
return typ
return provides
class IBar(Interface):
"""Declare something about IBar here"""
@provides(IBar)
class Foo(object):
"""Implement something here..."""
.. _PyProtocols: http://peak.telecommunity.com/PyProtocols.html
Of course, all these examples are possible today, though without
syntactic support.
Open Issues
===========
1. It's not yet certain that class decorators will be incorporated
into the language at this point. Guido expressed skepticism about
the concept, but various people have made some `strong arguments`_
(search for ``PEP 318 -- posting draft``) on their behalf in
``python-dev``.
.. _strong arguments:
http://mail.python.org/pipermail/python-dev/2004-March/thread.html
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
More information about the Python-Dev
mailing list