[Python-ideas] Change how Generator Expressions handle StopIteration

[Ron Adam]

On 11/06/2014 04:15 AM, Steven D'Aprano wrote:
> The current situation is simple to learn and understand:
>
> (1) Generator expressions*emit*  StopIteration when they are done:
>
> py> next(iter([]))
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> StopIteration

It's the "when they are done" part that's having the issue.  In some cases, 
they are never completely done because the StopIteration Error is handled 
incorrectly.  The reason this doesn't show up more is that most iterators 
only have one loop, which is exited and the generator ends causing a 
different StopIteration to be emitted from the one that ends the loop.

> (2) Functions such as tuple, list, set, dict*absorb*  StopIteration:
>
> py> list(iter([]))
> []
> py> it = iter([])
> py> list(next(it) for y in range(1000))
> []

Right, sometimes this doesn't happen.


> For-loops do the same, if StopIteration is raised in the "for x in
> iterable" header. That's how it knows the loop is done. The "for" part
> of a comprehension is the same.

I think this part is working as it should.

> (3) But raising StopIteration in the expression part (or if part) of a
> comprehension does not absord the exception, it is treated like any
> other exception:
>
> py> [next(iter([])) for y in range(1000)]
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "<stdin>", line 1, in <listcomp>
> StopIteration

This is not always working.  When a StopIteration is raised from the next 
call in the example that started the thread, it's getting replaced with the 
equivalent of a break, so it's never exiting the generator completely.  I 
think it needs to be replaced with the equivalent of return, which will end 
the generator, and cause a StopIteration to be emitted.  (As you describe 
here.)

It looks like it's a bug in the C code for generator expressions.  Since it 
only effects generator expressions that have iterators with nested loops, 
it may be fixible.

Cheers,
    Ron