TL;DR :-(

The crux of the issue is that sometimes the StopIteration raised by an explicit next() call is "just" an exception, but sometimes it terminates some controlling loop. Most precedents clearly establish it as "just" an exception, e.g. (these are all nonsense)

it = iter([1, 2, 3, -1])

for x in it:

if x < 0:

next(it)

it = iter([1, 2, 3])

[x for x in it if next(it) < 0]

Both these raise StopIteration and report a traceback.

But generators and generator expressions are different. In a generator, any StopIteration that isn't caught by an exception handler in the body of the generator implicitly terminates the iteration, just like "return" or falling off the end. This was an explicitly designed feature, but I don't think has worked out very well, given that more often than not, a StopIteration that "escapes" from some expression is a bug.

And because generator expressions are implemented using generators (duh), the following returns [] instead of raising StopIteration:

it = iter([1, 2, 3])

list(x for x in it if next(it) < 0)

This is confusing because it breaks the (intended) equivalence between list(<genexp>) and [<genexp>] (even though we refer to the latter as a comprehension, the syntax inside the [] is the same as a generator expression.

If I had had the right foresight, I would have made it an error to terminate a generator with a StopIteration, probably by raising another exception chained to the StopIteration (so the traceback shows the place where the StopIteration escaped).

The question at hand is if we can fix this post-hoc, using clever tricks and (of course) a deprecation period.

--Guido

On Thu, Nov 6, 2014 at 2:15 AM, Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote:

> And having said that... what if we introduced UnexpectedStopIteration but
> initially made it a subclass of StopIteration?
>
> We could issue a deprecation warning whenever we triggered the
> StopIteration -> UnexpectedStopIteration conversion, pointing out that at
> some point in the future (3.6? 3.7?), UnexpectedStopIteration will no
> longer be a subclass of StopIteration (perhaps becoming a subclass of
> RuntimeError instead?).

I'm sorry, I have been trying to follow this thread, but there have
been too many wrong turns and side-tracks for me to keep it straight.
What is the problem this is supposed to solve?

Is it just that list (and set and dict) comprehensions treat
StopIteration differently than generator expressions? That is, that

[expr for x in iterator]

list(expr for x in iterator)

are not exactly equivalent, if expr raises StopIteration.

If so, it seems to me that you're adding a lot of conceptual baggage and
complication for very little benefit, and this will probably confuse
people far more than the current situation does. The different treatment
of StopIteration in generator expressions and list comprehensions does
not seem to be a problem for people in practice, judging by the
python-list and tutor mailing lists.

The current situation is simple to learn and understand:

(1) Generator expressions *emit* StopIteration when they are done:

py> next(iter([]))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

(2) Functions such as tuple, list, set, dict *absorb* StopIteration:

py> list(iter([]))
[]
py> it = iter([])
py> list(next(it) for y in range(1000))
[]

For-loops do the same, if StopIteration is raised in the "for x in
iterable" header. That's how it knows the loop is done. The "for" part
of a comprehension is the same.

(3) But raising StopIteration in the expression part (or if part) of a
comprehension does not absord the exception, it is treated like any
other exception:

py> [next(iter([])) for y in range(1000)]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <listcomp>
StopIteration

If that is surprising to anyone, I suggest it is because they haven't
considered what happens when you raise StopIteration in the body of a
for-loop:

py> for y in range(1000):
... next(iter([]))
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
StopIteration

To me, the status quo is consistent, understandable, and predictable.

In contrast, you have:

- a solution to something which I'm not sure is even a problem
that needs solving;

- but if it does, the solution seems quite magical, complicated,
and hard to understand;

- it is unclear (to me) under what circumstances StopIteration
will be automatically converted to UnexpectedStopIteration;

- and it seems to me that it will lead to surprising behaviour
when people deliberately raise StopIteration only to have it
mysteriously turn into a different exception, but only
sometimes.

It seems to me that if the difference between comprehensions and
generator expressions really is a problem that needs solving, that the
best way to proceed is using the __future__ mechanism. 3.5 could
introduce

from __future__ comprehensions_absorb_stopiteration

and then 3.6 or 3.7 could make it the default behaviour.

We're still breaking backwards compatibility, but at least we're doing
it cleanly, without magic (well, except the __future__ magic, but that's
well-known and acceptible magic). There will be a transition period
during which people can choose to keep the old behaviour or the new, and
then we transition to the new behaviour. This automatic transformation
of some StopIterations into something else seems like it will be worse
than the problem it is trying to fix.

For what it is worth, I'm a strong -1 on changing the behaviour of
comprehensions at all, but if we must change it in a backwards
incompatible way, +1 on __future__ and -1 on changing the exceptions to
a different exception.

--
Steven

_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--Guido van Rossum (python.org/~guido)