PEP 288 ponderings

[Michael Sparks]

On Sun, 2 Jan 2005, Ian Bicking wrote:

> Steven Bethard wrote:
> > PEP 288 was mentioned in one of the lambda threads and so I ended up
> > reading it for the first time recently.  I definitely don't like the
> > idea of a magical __self__ variable that isn't declared anywhere.  It
> > also seemed to me like generator attributes don't really solve the
> > problem very cleanly.  An example from the PEP[1]:
> >
> >     def mygen():
> >         while True:
> >             print __self__.data
> >             yield None
> >
> >     g = mygen()
> >     g.data = 1
> >     g.next()                # prints 1
> >     g.data = 2
> >     g.next()                # prints 2
>
> I don't get why this isn't good enough:
>
>      def mygen(data):
>          while True:
>              print data[0]
>              yield None

It's OK, but rather limited. IMHO a nicer approach is to use decorators to
decorate a generator with extra attributes. Specifically to embed the
generator inside an iterator inside an anonymous class. (We're exploring
this as a alternative approach to the system we currently use at work)
(Hmm... Now I describe it that sounds rather hideous, but the result is
nice)

First of all the decorator:

import copy
def wrapgenerator(bases=object, **attrs):
   def decorate(func):
       class statefulgenerator(bases):
          __doc__ = func.__doc__
          def __init__(self,*args):
             super(statefulgenerator, self).__init__(*args)
             self.func=func(self,*args)
             for k in attrs.keys(): self.__dict__[k] = copy.deepcopy(attrs[k])
             self.next=self.__iter__().next
          def __iter__(self): return iter(self.func)
       return statefulgenerator
   return decorate

This would be used thus:

@wrapgenerator(component)
def forwarder(self):
   # The generator, note the explicit self for local state
   # for this generator
   "Simple data forwarding generator"
   while 1:
      if self.dataReady("inbox"):
         self.send("outbox",self.recv("inbox"))
      elif self.dataReady("control"):
         if self.recv("control") == "shutdown":
            break
      yield 1
   self.send("signal","shutdown")
   yield 0

This clearly assumes a set of attributes, local methods etc.

In this case our local methods, attribute etc are from this class:
class component(object):
   def __init__(self, *args):
       # Default queues
       self.queues = {"inbox":[],"control":[],"outbox":[],"signal":[]}
   def send(self, box, object): self.queues[box].append(object)
   def dataReady(self,box): return len(self.queues[box])>0
   def recv(self, box): # NB. Exceptions aren't caught
      X=self.queues[box][0]
      del self.queues[box][0]
      return X

This then gets used in the usual way:
>>> f = forwarder()
>>> f
<__main__.statefulgenerator object at 0x402dcccc>
>>> f.send("inbox", "some data")
>>> f.send("inbox", "some more data")
>>> f.next()
1
>>> f.next()
1
>>> f.recv("outbox")
'some data'
>>> f.recv("outbox")
'some more data'

If you "just" want (arbitrary) generator attributes, then that's a lot
easier (since we don't need to support an arbitrary base class):

import copy
def add_attrs_to_generator(**attrs):
   def decorate(func):
       class statefulgenerator:
          __doc__ = func.__doc__
          def __init__(self,*args):
             self.func=func(self,*args)
             for k in attrs.keys(): self.__dict__[k] = copy.deepcopy(attrs[k])
             self.next=self.__iter__().next
          def __iter__(self): return iter(self.func)
       return statefulgenerator
   return decorate

@add_attrs_to_generator()
def simpleGen(self):
   while True:
      print dir(self)
      yield None

We can also add default attributes:

@add_attrs_to_generator(test=1,simple=2)
def simpleGen(self):
   while True:
      print dir(self)
      yield None

>>> g=simpleGen()
>>> g.next()
['__doc__', '__init__', '__iter__', '__module__', 'func', 'next',
'simple', 'test']

We've been using this general approach of giving generators local state
for communications for a while now, and it's been rather interesting/fun
to use. However we've been using explicit declaration of a class, and then
a standard fixed name for the generator (main) instead. (we've been
thinking about using decorators as syntactic sugar)

Being able to pass exception in would be very nice, however we've used a
"control"  attribute, queue in practice, to allow for need to be able to
shutdown an arbitrary generator. However, I don't think the __self__ part
of the PEP is explicitly required - since you can build that onto
generators using python quite happily now (@add_attrs_to_generator).

The code for the stuff we're playing with (in case anyone's interested :)
is at http://kamaelia.sourceforge.net/. The specific subsystem that uses
this approach has been released as a separate download (Axon[1]) on the
sourceforge project page[2], and the main system (obviously) uses this
heavily and can be browsed via viewcvs. [3]
   [1] http://kamaelia.sourceforge.net/Docs/Axon.html
   [2] http://sourceforge.net/projects/kamaelia
   [3] http://cvs.sourceforge.net/viewcvs.py/kamaelia/Code/Python/Kamaelia/Kamaelia/

Best Regards,


Michael.