[Python-ideas] Propagating StopIteration value
tjreedy at udel.edu
Tue Oct 9 23:37:46 CEST 2012
On 10/9/2012 11:34 AM, Serhiy Storchaka wrote:
> On 09.10.12 10:51, Greg Ewing wrote:
>> Where we seem to disagree is on
>> whether returning a value with StopIteration is part of the
>> iterator protocol or the generator protocol.
There is a generator class but no 'generator protocol'. Adding the extra
generator methods to another iterator class will not give its instances
the suspend/resume behavior of generators. That requires the special
bytecodes and flags resulting from the presence of 'yield' in the
generator function whose call produces the generator.
> Is a generator expression work with the iterator protocol or the
> generator protocol?
A generator expression produces a generator, which implements the
iterator protocol and has the extra generator methods and suspend/resume
Part of the iterator protocol is that .__next__ methods raise
StopIteration to signal that no more objects will be yielded. A value
can be attached to StopIteration, but it is irrelevant to it use as a
'done' signal. Any iterator .__next__ method. can raise or pass along
StopIteration(something). Whether 'something' is even seen or not is a
different question. The main users of iterators, for statements, ignore
> A generator expression eats a value with StopIteration:
> >>> def G():
> ... return 42
> ... yield
> >>> next(x for x in G())
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> Is it a bug?
Of course not. A generator expression is an abbreviation for a def
statement defining a generator function followed by a call to that
generator function. (x for x in G()) is roughly equivalent to
for x in G():
# when execution reaches here, None is returned, as usual
_ = __()
_ # IE, _ is the value of the expression
A for loop stops when it catches (and swallows) a StopIteration
instance. That instance has served it function as a signal. The for
mechanism ignores any attributes thereof.
The generator .__next__ method that wraps the generator code object (the
compiled body of the generator function) raises StopIteration if the
code object ends by returning None. So the StopIteration printed in the
traceback above is a different StopIteration instance and come from a
different callable than the one from G that stopped the for loop in the
generator. There is no sensible way to connect the two. Note that a
generator can iterate through multiple iterators, like map and chain do.
If the generator stops by raising StopIteration instead of returning
None, *that* StopIteration instance is passed along by the .__next__
wrapper. (This may be an implementation detail, but it is currently true.)
>>> def g2():
SI = StopIteration('g2')
>>> try: next(g2())
except StopIteration as SI:
If you want any iterator to raise or propagate a value-laden
StopIteration, you must do it explicitly or avoid swallowing one.
>>> def G(): return 42; yield
>>> def g3(): # replacement for your generator expression
it = iter(G())
Traceback (most recent call last):
File "<pyshell#29>", line 1, in <module>
File "<pyshell#28>", line 4, in g3
StopIteration: 42 # now you see the value
Since filter takes a single iterable, it can be written like g3 and not
catch the StopIteration of the corresponding iterator.
def filter(pred, iterable):
it = iter(iterable)
item = next(it)
# never reaches here, never returns None
Map takes multiple iterables. In 2.x, map extended short iterables with
None to match the longest. So it had to swallow StopIteration until it
had collected one for each iterator. In 3.x, map stops at the first
StopIteration, so it probably could be rewritten to not catch it.
Whether it makes sense to do that is another question.
Terry Jan Reedy
More information about the Python-ideas