On Thu, Sep 25, 2014 at 2:50 AM, Nathaniel Smith njs@pobox.com wrote:
The most elegant solution I know is:
class PushbackAdaptor: def __init__(self, iterable): self.base = iter(iterable) self.stack = []
def next(self): if self.stack: return self.stack.pop() else: return self.base.next() def pushback(self, obj): self.stack.append(obj)it = iter(character_source) for char in it: ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN it.push_back(char) continue ...
In modern python, I think the natural meaning for 'continue with' wouldn't be to special-case something like this. Instead, where 'continue' triggers a call to 'it.next()', I'd expect 'continue with x' to trigger a call to 'it.send(x)'. I suspect this might enable some nice idioms in coroutiney code, though I'm not very familiar with such.
In fact, given the 'send' definition of 'continue with x', the above tokenization code would become simply:
def redoable(iterable): for obj in iterable: while yield obj == "redo": pass
for char in redoable(character_source): ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN continue with "redo" ...
which I have to admit is fairly sexy.