PEP 288: Generator Attributes

[Bengt Richter]

On 4 Dec 2002 19:34:30 -0800, logistix at zworg.com (logistix) wrote:

>What advantage do Generator Attributes have over a class that
>implements the iterator interface?  I always assumed that generators
>were just shorthand for an iterable (if thats a word) class.
>

A generator remembers a different kind of state between calls to .next().
The class iterable remembers whatever attributes you attach to "self"
and that's it (other than globals and special stuff like default method
parameter values etc).

But a generator is like a function that remembers its local variables and
also its state of execution control at special points (namely where
the yield statements are). It remembers where it was at the last yield
that it executed to return a value. The call to .next() doesn't just
execute a defined method like the one in your class foo -- it continues
on the line after the last yield. That's a big difference!! It means you
can return a value from inside a loop and continue on, e.g., consider
a simple state machine parsing a string into a sequence of names and numbers:
(Note that "names" can have trailing digits here, and numbers are unsigned integers)

 >>> from __future__ import generators
 >>> def namenum(s):
 ...     state = '?'
 ...     sl = list(s); sl.reverse()
 ...     while sl:
 ...         c = sl.pop()
 ...         if state=='?':
 ...             if c.isspace(): continue
 ...             if c.isdigit(): state = 'N'; n = int(c)
 ...             else: state = 'T'; sout = [c]
 ...         elif state == 'N':
 ...             if not c.isdigit() or not sl:
 ...                 yield n
 ...                 state = '?'; sl.append(c) # reuse non-digit
 ...             else:
 ...                 n = n*10 + int(c)
 ...         else: #T
 ...             if not c.isspace(): sout.append(c)
 ...             if c.isspace() or not sl:
 ...                 yield ''.join(sout)
 ...                 state = '?'
 ...
 >>> g = namenum('  hi ho 123 4 five5 6six see?')
 >>> [x for x in g]
 ['hi', 'ho', 123, 4, 'five5', 6, 'six', 'see?']
 >>>

The above could be implemented as a class supporting iter, but
it wouldn't be quite as easy to code. It could probably be
done as a one-line list comprehension too, but it's nice to
have more than one way to do it (;-) so you can choose one
that expresses the semantics best (to a human).

>>>> class foo:
>... 	def __iter__(self):
>... 		return self
>... 	def next(self):
>... 		print self.data
>... 	def __init__(self):
>... 		self.data = 1
>... 		
>>>> x = iter(foo())
>>>> x.next()
>1
>>>> x.next()
>1
>>>> x.data = 3
>>>> x.next()
>3
>>>> y = iter(foo())
>>>> y.next()
>1
>
>
>Of course Generator Attributes would also be shorthand, but I don't
>think it'd be good shorthand.  It'd allow you to do many unobvious
>things with your generators.

Passing parameters into a running generator or iterator is a special concern
that should not obscure the other important differences and commonalities, IMO.

Regards,
Bengt Richter