Using metaclasses to play with decorators.

Michael Sparks zathras at thwackety.com
Tue Jun 15 02:04:30 CEST 2004


[ I'm not really sure of the etiquette of the python-dev list, so I think
  I'll play "safe" and post this thought here... I know some of the
  developers do look here, and Guido's comment (before europython) on the
  dev list saying he's not interested in new syntaxes makes me think this
  is a better place for it... ]

Anyway...

At Europython Guido discussed with everyone the outstanding issue with
decorators and there was a clear majority in favour of having them, which
was good. From where I was sitting it looked like about 20:20 split on the
following syntaxes:
1    def func(arg1, arg2, arg3) [dec,dec,dec]:
        function...
2    [dec,dec,dec] def func(arg1, arg2, arg3):
        function...

When it came to the decision on whether 2.4 includin one of these two
options or a waiting for a third unspecified, indeterminate syntax in a
later version, it looked like around a 60:40 vote in favour of waiting.
(ie 60:20:20 for waiting/syntax 1/syntax 2)

Also, suggested use cases for decorators were:
   * Changing function type from method to staticmethod, classmethod
   * Adding metadata:
      * Author
      * Version
      * Deprecation
   * Processing rules  - eg grammar rules
   * Framework annotations
   * Support for external bindings

Some of these are clearly transformations of the methods, some are
annotations - ie addition of attributes after creation.

Transformations of methods can already be done using metaclasses.

Attributes can already appended to methods, but these need to be done
before transformations, and have to be done after the function has been
created:

class foo(object):
   def bar(hello):
       "This is"
       print hello
   bar.__RULE__ = "expr := expr"
   bar.__doc__ += " a test"
   bar.__author__ = "John"
   bar = staticmethod(bar)

Suppose for a moment we wanted to do the same thing for the class foo - ie
we had some attributes we wanted to add to it, and after creating it we
wanted to transform it - what might we write?

class foo(object):
   __RULE__ = "expr := expr"
   __doc__ += " a test"
   __author__ = "John"
   def bar(hello):
       "This is"
       print hello

foo = someclasstransform(foo)

That's a lot nicer. However if we treat foo as a class description, then
someclasstransform tends to look similar to metaclass shenanigans.
Does that mean we can use metaclasses to similate a syntax for things
like staticmethods ?

My opinion here is yes - it's relatively trivial to implement if you use a
naming
scheme for methods:
class decorated_class_one(type):
     def __new__(cls, name, bases, dct):
         for key in dct.keys():
            if key[:12] == "classmethod_":
               dct[ key[12:] ] = classmethod(dct[key])
               del dct[key]
            if key[:7] == "static_":
               dct[ key[7:] ] = staticmethod(dct[key])
               del dct[key]
         return type.__new__(cls, name, bases, dct)

class myclass(object):
   __metaclass__ = decorated_class_one
   def static_foo(arg1, arg2, arg3):
      print "Hello",arg1, arg2, arg3
   def classmethod_bar(cls,arg1, arg2, arg3):
      print  "World",cls, arg1, arg2, arg3
   def baz(self,arg1, arg2, arg3):
      print  "There",self, arg1, arg2, arg3

The question then becomes what's the cleanest way of adding attributes
using this approach?

Since we've got a syntax that's similar to classes, you might argue one
approach might be:
class myclass(object):
   __metaclass__ = decorated_class
   def static_foo(arg1, arg2, arg3):
      __doc__ = "This is a static method"
      __author__ = "Tom"
      __deprecated__ = True
      print "Hello",arg1, arg2, arg3
   def classmethod_bar(cls,arg1, arg2, arg3):
      __doc__ = "This is a class method"
      __author__ = "Dick"
      __deprecated__ = False
      print  "World",cls, arg1, arg2, arg3
   def baz(self,arg1, arg2, arg3):
      __doc__ = "This is a normal method"
      __author__ = "Harry"
      __deprecated__ = False
      print  "There",self, arg1, arg2, arg3

Whilst we can get at the variable names that these declare, we can't get
at the values. What else can we do?

We could choose a similar "meta" keyword - say "decorators_", and pass a
dictionary instead?

class myclass(object):
   __metaclass__ = decorated_class_two
   decorators_static_foo = {
      '__doc__' : "This is a static method",
      '__author__' : "Tom",
      '__deprecated__' : True
   }
   def static_foo(arg1, arg2, arg3):
      print "Hello",arg1, arg2, arg3
   decorators_classmethod_bar = {
      '__doc__' : "This is a class method",
      '__author__' : "Dick",
      '__deprecated__' : False
   }
   def classmethod_bar(cls,arg1, arg2, arg3):
      print  "World",cls, arg1, arg2, arg3
   decorators_baz = {
      '__doc__' : "This is a normal method",
      '__author__' : "Harry",
      '__deprecated__' : False
   }
   def baz(self,arg1, arg2, arg3):
      print  "There",self, arg1, arg2, arg3

Whilst it's not perfect, it's something we can actually use. Bear in mind
we have to add attributes/decorators before we do transforms:

class decorated_class_two(type):
     def __new__(cls, name, bases, dct):
         for key in dct.keys():
            if key[:11] == "decorators_":
               for attr_key in dct[key].keys():
                  exec 'dct[key[11:]].'+attr_key+' = dct[key][attr_key]'
               del dct[key]
         for key in dct.keys():
            if key[:12] == "classmethod_":
               dct[ key[12:] ] = classmethod(dct[key])
               del dct[key]
            if key[:7] == "static_":
               dct[ key[7:] ] = staticmethod(dct[key])
               del dct[key]
         return type.__new__(cls, name, bases, dct)

Can we go one better? Can we make the following work?
class myclass(object):
   __metaclass__ = decorated_class_three
   decorators_foo = {
      'transforms' : [staticmethod],
      '__doc__' : "This is a static method",
      '__author__' : "Tom",
      '__deprecated__' : True
   }
   def foo(arg1, arg2, arg3):
      print "Hello",arg1, arg2, arg3
   decorators_bar = {
      'transforms' : [classmethod],
      '__doc__' : "This is a class method",
      '__author__' : "Dick",
      '__deprecated__' : False
   }
   def bar(cls,arg1, arg2, arg3):
      print  "World",cls, arg1, arg2, arg3
   decorators_baz = {
      '__doc__' : "This is a normal method",
      '__author__' : "Harry",
      '__deprecated__' : False
   }
   def baz(self,arg1, arg2, arg3):
      print  "There",self, arg1, arg2, arg3

Well, let's try:
class decorated_class_three(type):
     def __new__(cls, name, bases, dct):
         for key in dct.keys():
            if key[:11] == "decorators_":
               transforms = []
               for attr_key in dct[key].keys():
                  if attr_key == 'transforms':
                     transforms = dct[key][attr_key]
                     continue
                  exec 'dct[key[11:]].'+attr_key+' = dct[key][attr_key]'
               for transform in transforms:
                  dct[key[11:]] = transform(dct[key[11:]])
               del dct[key]
         return type.__new__(cls, name, bases, dct)

And hey presto! It works!

The upshot is is that using a very simple metaclass, we can already have
the functionality that decorators will give us, but the syntax is less
than ideal:

class myclass(object):
   __metaclass__ = decorated_class_three
   decorators_foo = {
      'transforms' : [staticmethod],
      '__doc__' : "This is a static method",
      '__author__' : "Tom",
      '__deprecated__' : True
   }
   def foo(arg1, arg2, arg3):
      print "Hello",arg1, arg2, arg3

Now let's simplify this to what people currently commonly do:
class myclass(object):
   def foo(arg1, arg2, arg3):
     "This is a static method"
      print "Hello",arg1, arg2, arg3
   foo = staticmethod(foo)

In this situation python *does* do something special with the first value
it finds as the first value inside the method - it uses it as a __doc__
decorator. To my mind this treating the first value of the code as special
strikes me as the ideal hook. You could, for example, have the following
syntax:

class myclass(object):
   def foo(arg1, arg2, arg3):
     { 'transforms' : [staticmethod],
       '__doc__' : "This is a static method",
       '__author__' : "Tom",
       '__deprecated__' : True
     }
     print "Hello",arg1, arg2, arg3

But here's the really cool trick - we can actually use this. Now. Let's
put our metaclass back, and make a small modification to make this work:
class myclass(object):
   __metaclass__ = decorated_class_four
   def foo(arg1, arg2, arg3):
     """{ 'transforms' : [staticmethod],
       '__doc__' : "This is a static method",
       '__author__' : "Tom",
       '__deprecated__' : True
     }"""
     print "Hello",arg1, arg2, arg3

We can then modify our metaclass to make this work:
class decorated_class_four(type):
     def __new__(cls, name, bases, dct):
         for key in dct.keys():
            doc = dct[key].__doc__
            if doc:
               try:
                  transforms = []
                  decorators = eval(doc)
                  for attr_key in decorators:
                     if attr_key == 'transforms':
                        transforms = decorators[attr_key]
                        continue
                     exec 'dct[key].'+attr_key+' = decorators[attr_key]'
                  for transform in transforms:
                     dct[key] = transform(dct[key])
               except SyntaxError:
                  pass # no decorators
         return type.__new__(cls, name, bases, dct)

So recasting our example using this final syntax, we gain:
class myclass(object):
   __metaclass__ = decorated_class_four

   def foo(arg1, arg2, arg3):
      """{ 'transforms' : [staticmethod],
           '__doc__' : "This is a static method",
           '__author__' : "Tom",
           '__deprecated__' : True
      }"""
      print "Hello",arg1, arg2, arg3

   def bar(cls,arg1, arg2, arg3):
      """{ 'transforms' : [classmethod],
           '__doc__' : "This is a class method",
           '__author__' : "Dick",
           '__deprecated__' : False
      }"""
      print  "World",cls, arg1, arg2, arg3

   def baz(self,arg1, arg2, arg3):
      """{ '__doc__' : "This is a normal method",
           '__author__' : "Harry",
           '__deprecated__' : False
         }"""
      print  "There",self, arg1, arg2, arg3

Each of the transformations chosen takes us between a variety of syntaxes,
with various advantages/disadvantages. Personally I think the best variety
here without changing python's syntax or semantics is this one:

class myclass(object):
   __metaclass__ = decorated_class_three
   decorators_foo = {
      'transforms' : [staticmethod],
      '__doc__' : "This is a static method",
      '__author__' : "Tom",
      '__deprecated__' : True
   }
   def foo(arg1, arg2, arg3):
      print "Hello",arg1, arg2, arg3
...

Partly the reason for this is because it's very clear  what's going on
here, and also you can use doc strings as normal:
class myclass(object):
   __metaclass__ = decorated_class_three
   decorators_foo = {
      'transforms' : [staticmethod],
      '__author__' : "Tom",
      '__deprecated__' : True
   }
   def foo(arg1, arg2, arg3):
      "This is a static method"
      print "Hello",arg1, arg2, arg3
...

Whereas with a semantic change to the initial var, I think this form is
nicer:
class myclass(object):
   # no metaclass, requires change in semantics
   def foo(arg1, arg2, arg3):
      { 'transforms' : [staticmethod],
        '__doc__' : "This is a static method",
        '__author__' : "Tom",
        '__deprecated__' : True
      }
      print "Hello",arg1, arg2, arg3

   def bar(cls,arg1, arg2, arg3):
      { 'transforms' : [classmethod],
        '__doc__' : "This is a class method",
        '__author__' : "Dick",
        '__deprecated__' : False
      }
      print  "World",cls, arg1, arg2, arg3
...


Syntactically this compiles/runs fine right now, it just doesn't have the
decorator semantics we need for this to work.

And what would the semantic change be? Currently we have foo.__doc__ .
This could be foo.__decorator_dict__ .  Quite what anyone chooses to do
with these would be entirely up to them.

Classes that inherit from object could be defined to do something special
- such as act in a similar way to the presented metaclass. Whereas for
objects that aren't derived from object, nothing would change, except a
small amount of extra information is made available, and *potentially*
ignored.

Anyway, for any of the python-dev team who do hang on out python-list I
hope this has been food for thought, and whilst I've done a cursory check
of the archives I'm not subbed to python-dev, so apologies if I'm going
over old ground and raking up old ideas needlessly!

One thing that does strike me regarding this is this:

This relies on being able to pull out the first value at the start of the
function/method definition. This _does_ currently happen anyway, and
people are using the value there. Making it consistent in that you are
able to pull out the value, no matter what the type strikes me as a very
useful thing, and decorators can naturally fall out of it as well.

As I said, hopefully useful food for thought, and hopefully not hacked
anyone off at the length of this post!

Cheers,


Michael.





More information about the Python-list mailing list