[Python-ideas] Change how Generator Expressions handle StopIteration
Terry Reedy
tjreedy at udel.edu
Sun Nov 2 00:07:08 CET 2014
On 11/1/2014 12:50 PM, Guido van Rossum wrote:
> I think you're on to something. But I think both your examples have a
> problem, even though your second one "works".
Both versions are buggy in that iizip() yields () infinitely, while
zip() yields nothing. Fixes below.
> If we weren't forced by backward compatibility I would have made it much
> harder for StopIteration to "leak out". Currently a generator can either
> return or raise StopIteration to signal it is done, but I think it would
> have been better if StopIteration was treated as some kind of error in
> this case.
This would require some sort of additional special casing of
StopIteration that we do not have now. Currently, it is limited to
'for' loops expecting and catching StopIteration as a signal to stop
iterating. That is rather easy to understand.
> Basically I think any time a StopIteration isn't caught by a
> for-loop or an explicit try/except StopIteraton, I feel there is a bug
> in the program,
Outside of generator functions (and expressions), I agree as I cannot
think of an exception when it is not. This has come up on Python list.
> or at least it is hard to debug.
Code within generator functions is different. Writing "raise
StopIteration" instead of "return" is mostly a waste of keystrokes. As
for next(it), StopIteration should usually propagate, as with an
explicit raise and not be caught. The code below that 'works' (when it
does work), works because the StopIteration from next(it) (when there is
at least one) propagates to the list comp, which lets it pass to the
generator, which lets it pass to the generator user.
> I'm afraid that ship has sailed, though...
>
> On Sat, Nov 1, 2014 at 7:56 AM, yotam vaknin
> <tomirendo at gmail.com
> <mailto:tomirendo at gmail.com>> wrote:
> I would like to purpose that generator expressions will not catch
> StopIteration exception, if this exception did not come from the
> iterated object's __next__ function specifically.
For the purpose of your example, all instances of StopIteration are the
same and might as well be the same instance.
Since to my understanding generators and g.e.s already do not catch the
StopIterations you say you want not caught, and since you need for it to
not be caught in the code below, I do not understand exactly what you
propose.
> So generator
> expressions will be able to raise StopIteration by calculating the
> current value of the Generator.
I cannot understand this.
> def izip(*args):
> iters = [iter(obj) for obj in args]
> while True:
> yield tuple(next(it) for it in iters)
>
> a = izip([1,2],[3,4])
> print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) ()
> list(izip([1,2],[3,4])) #Currently never returns
Better test code that avoid infinite looping:
a = izip([1,2],[3,4])
for i in range(3):
print(next(a))
One the third loop, the above prints (), while the below prints a
traceback. With a = izip(), both print () 3 times.
The problem is that when next(it) raises, you want the StopIteration
instance propagated (not immediately caught), so that the
generator-using code knows that the generator is exhausted. But the
tuple call catches it first, so that, in combination with 'while True',
the user never sees StopIteration A partial solution is to provoke
StopIteration before calling tuple, so that it does propagate. That is
what the list comp below does. But if args is empty, so is iters, and
there is no next(it) to ever raise. For a complete solution that
imitates zip and does not require an otherwise useless temporary list,
replace the loop with this:
while True:
t = tuple(next(it) for it in iters)
if not t:
return
yield t
> Even thought this is the PEP described behaviour, I think this is an
> unwanted behaviour.
Not if you think carefully about what you want to happen when next(it)
raises. I think generators and generators expressions should be left
alone.
> I think Generator Expressions should work like List Comprehension in
> that sense:
>
> def iizip(*args):
> iters = [iter(obj) for obj in args]
> while True:
> yield tuple([next(it) for it in iters])
This could be fixed with 'if not iters: return' as the second line.
Replacing [genexp] with list(genexp) does not work because the latter,
unlike the former, catches StopIteration. This is proof that the two
are not exactly equivalent, and the such behavior difference I know of
(excluding introspection, such as with trace).
--
Terry Jan Reedy
More information about the Python-ideas
mailing list