keeping state in an iterator object by rebinding next()

[Peter Otten]

Wilbert Berendsen wrote:

> Hi,
> 
> i am writing a simple parser, that generates tokens. The parser needs to
> maintain some state, because some parts of the file consist of different
> tokens. I thought the object could simply remember its state by assigning
> it's next() method to the method that is currently parsing. When the state
> changes, the called method rebinds next() and the next token will be
> returned by that function. Here's an example, proving that this indeed
> works.
> 
>>>> class A:
> ...  def a(self):
> ...   self.next = self.b
> ...   return 1
> ...  def b(self):
> ...   self.next = self.a
> ...   return 2
> ...  def __iter__(self):
> ...   return self
> ...
>>>> a=A()
>>>> a.a()
> 1
>>>> a.next()
> 2
>>>> a.next()
> 1
>>>> j=0
>>>> for i in a:
> ...  j += 1
> ...  if j > 10: break # prevent from running endlessly
> ...  print i
> ...
> 2
> 1
> 2
> 1
> 2
> 1
> 2
> 1
> 2
> 1
>>>>
> 
> my question is: is this legal Python? An iterator could save the next()
> method object, and in that case it could stop working.... It works now,
> because apparently the for- construct resolves 'next' each time for the
> object before calling it.
> 
> The other solution would be just jumping to the correct method from within
> the next() method. But that gives an extra call...

It doesn't work with newstyle classes though as next() is looked up in the
class rather than the instance, just like the __xxx__() methods:

>>> class A(object):
...     def __iter__(self): return self
...     def next(self): return "class-next()"
...
>>> a = A()
>>> a.next = lambda: "instance-next()"
>>> a.next()
'instance-next()'
>>> from itertools import islice
>>> for item in islice(a, 3): print item
...
class-next()
class-next()
class-next()

Without the class-next() it isn't even recognized as an iterator

>>> del A.next
>>> for item in a: pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: iter() returned non-iterator of type 'A'

even though the instance-next() is still there:

>>> a.next()
'instance-next()'

So I recommend that you avoid that can of worms and go with the extra
indirection.

Peter