On 10/9/2012 11:34 AM, Serhiy Storchaka wrote:
On 09.10.12 10:51, Greg Ewing wrote:
Where we seem to disagree is on whether returning a value with StopIteration is part of the iterator protocol or the generator protocol.
There is a generator class but no 'generator protocol'. Adding the extra generator methods to another iterator class will not give its instances the suspend/resume behavior of generators. That requires the special bytecodes and flags resulting from the presence of 'yield' in the generator function whose call produces the generator.
Is a generator expression work with the iterator protocol or the generator protocol?
A generator expression produces a generator, which implements the iterator protocol and has the extra generator methods and suspend/resume behavior. Part of the iterator protocol is that .__next__ methods raise StopIteration to signal that no more objects will be yielded. A value can be attached to StopIteration, but it is irrelevant to it use as a 'done' signal. Any iterator .__next__ method. can raise or pass along StopIteration(something). Whether 'something' is even seen or not is a different question. The main users of iterators, for statements, ignore anything extra.
A generator expression eats a value with StopIteration:
def G(): ... return 42 ... yield ... next(x for x in G()) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
Is it a bug?
Of course not. A generator expression is an abbreviation for a def statement defining a generator function followed by a call to that generator function. (x for x in G()) is roughly equivalent to def __(): for x in G(): yield x # when execution reaches here, None is returned, as usual _ = __() del __ _ # IE, _ is the value of the expression A for loop stops when it catches (and swallows) a StopIteration instance. That instance has served it function as a signal. The for mechanism ignores any attributes thereof. The generator .__next__ method that wraps the generator code object (the compiled body of the generator function) raises StopIteration if the code object ends by returning None. So the StopIteration printed in the traceback above is a different StopIteration instance and come from a different callable than the one from G that stopped the for loop in the generator. There is no sensible way to connect the two. Note that a generator can iterate through multiple iterators, like map and chain do. If the generator stops by raising StopIteration instead of returning None, *that* StopIteration instance is passed along by the .__next__ wrapper. (This may be an implementation detail, but it is currently true.)
def g2(): SI = StopIteration('g2') print(SI, id(SI)) raise SI yield 1
try: next(g2()) except StopIteration as SI: print(SI, id(SI))
g2 52759816 g2 52759816 If you want any iterator to raise or propagate a value-laden StopIteration, you must do it explicitly or avoid swallowing one.
def G(): return 42; yield
def g3(): # replacement for your generator expression it = iter(G()) while True: yield next(it)
next(g3()) Traceback (most recent call last): File "<pyshell#29>", line 1, in <module> next(g3()) File "<pyshell#28>", line 4, in g3 yield next(it) StopIteration: 42 # now you see the value
Since filter takes a single iterable, it can be written like g3 and not catch the StopIteration of the corresponding iterator. def filter(pred, iterable): it = iter(iterable) while True: item = next(it) if pred(item): yield item # never reaches here, never returns None Map takes multiple iterables. In 2.x, map extended short iterables with None to match the longest. So it had to swallow StopIteration until it had collected one for each iterator. In 3.x, map stops at the first StopIteration, so it probably could be rewritten to not catch it. Whether it makes sense to do that is another question. -- Terry Jan Reedy