[Python-ideas] Change how Generator Expressions handle StopIteration

[Steven D'Aprano]

On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote:

> And having said that... what if we introduced UnexpectedStopIteration but
> initially made it a subclass of StopIteration?
> 
> We could issue a deprecation warning whenever we triggered the
> StopIteration -> UnexpectedStopIteration conversion, pointing out that at
> some point in the future (3.6? 3.7?), UnexpectedStopIteration will no
> longer be a subclass of StopIteration (perhaps becoming a subclass of
> RuntimeError instead?).

I'm sorry, I have been trying to follow this thread, but there have 
been too many wrong turns and side-tracks for me to keep it straight. 
What is the problem this is supposed to solve?

Is it just that list (and set and dict) comprehensions treat 
StopIteration differently than generator expressions? That is, that 

    [expr for x in iterator]

    list(expr for x in iterator)

are not exactly equivalent, if expr raises StopIteration.

If so, it seems to me that you're adding a lot of conceptual baggage and 
complication for very little benefit, and this will probably confuse 
people far more than the current situation does. The different treatment 
of StopIteration in generator expressions and list comprehensions does 
not seem to be a problem for people in practice, judging by the 
python-list and tutor mailing lists.

The current situation is simple to learn and understand:

(1) Generator expressions *emit* StopIteration when they are done:

py> next(iter([]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration


(2) Functions such as tuple, list, set, dict *absorb* StopIteration:

py> list(iter([]))
[]
py> it = iter([])
py> list(next(it) for y in range(1000))
[]

For-loops do the same, if StopIteration is raised in the "for x in 
iterable" header. That's how it knows the loop is done. The "for" part 
of a comprehension is the same.


(3) But raising StopIteration in the expression part (or if part) of a 
comprehension does not absord the exception, it is treated like any 
other exception:

py> [next(iter([])) for y in range(1000)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
StopIteration

If that is surprising to anyone, I suggest it is because they haven't 
considered what happens when you raise StopIteration in the body of a 
for-loop:

py> for y in range(1000):
...     next(iter([]))
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
StopIteration


To me, the status quo is consistent, understandable, and predictable.

In contrast, you have:

- a solution to something which I'm not sure is even a problem
  that needs solving;

- but if it does, the solution seems quite magical, complicated,
  and hard to understand;

- it is unclear (to me) under what circumstances StopIteration 
  will be automatically converted to UnexpectedStopIteration;

- and it seems to me that it will lead to surprising behaviour
  when people deliberately raise StopIteration only to have it
  mysteriously turn into a different exception, but only 
  sometimes.


It seems to me that if the difference between comprehensions and 
generator expressions really is a problem that needs solving, that the 
best way to proceed is using the __future__ mechanism. 3.5 could 
introduce

    from __future__ comprehensions_absorb_stopiteration

and then 3.6 or 3.7 could make it the default behaviour.

We're still breaking backwards compatibility, but at least we're doing 
it cleanly, without magic (well, except the __future__ magic, but that's 
well-known and acceptible magic). There will be a transition period 
during which people can choose to keep the old behaviour or the new, and 
then we transition to the new behaviour. This automatic transformation 
of some StopIterations into something else seems like it will be worse 
than the problem it is trying to fix.

For what it is worth, I'm a strong -1 on changing the behaviour of 
comprehensions at all, but if we must change it in a backwards 
incompatible way, +1 on __future__ and -1 on changing the exceptions to 
a different exception.



-- 
Steven