[Python-ideas] Change how Generator Expressions handle StopIteration

[Guido van Rossum]

On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 3 November 2014 13:01, Guido van Rossum <guido at python.org> wrote:
> > On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> >>
> >> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
> >>>
> >>> This is related to the fact that, although the docs imply otherwise,
> >>> [COMP] isn't exactly equivalent to list(COMP),
> >>
> >>
> >> That purported equivalence is a common meme, which I may have helped
> >> spread.
> >
> >
> > I may have started it. I was aware of the non-equivalence (only
> > mostly-equivalence) in Python 2 and I wanted to make then identical in
> > Python 3 -- having one construct being exactly equivalent to another
> reduce
> > the amount of explaining needed. Unfortunately, people had started to
> depend
> > on the (in my *current* opinion deplorable) behavior of generator
> > expressions in the face of StopIteration thrown by arbitrary parts of the
> > expression or condition, and the equivalence is still imperfect. At least
> > the variable leakage has been fixed.
>
> I think I'm guilty as well - when I was working on the Python 3
> changes, getting the *scoping* behaviour to be identical between
> comprehensions and generator expressions was one of the key
> objectives, so I regularly described it as making "[x for x in seq]"
> equivalent to "list(x for x in seq)".
>
> I unfortunately didn't notice the remaining exception handling
> differences at the time, or we might have been able to do something
> about it for 3.0 :(
>
> > However, I don't think this idea has panned out. I haven't done a survey,
> > but I have a feeling that in most cases where an explicit next() call is
> > used (as opposed to a for-loop) there's a try/except Stopiteration around
> > it, and a fair amount if time is wasted debugging situations where a
> > StopIteration unexpectedly escapes and silently interrupts some loop
> over an
> > unrelated generator (instead of loudly bubbling up to the top and
> causing a
> > traceback, which would be more debuggable). And the use case of raising
> > StopIteration from a condition used in a generator expression is iffy at
> > best (it makes the condition function hard to use in other contexts, and
> it
> > calls to attention the difference between generators and comprehensions).
> >
> > So I will go out on a limb here and say that this was a mistake and if we
> > can think of easing the transitional pain it would be a good thing to fix
> > this eventually.
>
> Having had to do the dance to work around the current behaviour in
> contextlib, I'm inclined to agree - there's a significant semantic
> difference between the "this iterable just terminated" StopIteration,
> and the "something threw StopIteration and nobody caught it", and the
> current model hides it.
>
> However, I also see potentially significant backwards compatibility
> problems when it comes to helper functions that throw StopIteration to
> terminate the calling generator - there would likely need to be some
> kind of thread local state and a helper "iterexit()" builtin and
> "PyIter_Exit()" C API to call instead of raising StopIteration
> directly.
>

That's what I was afraid of. Can you point me to an example of code that
depends on this that isn't trivial like Andrew Barnert's ensure_positive()
example? I think that particular example, and the category it represents,
are excessive cleverness that abuse the feature under discussion -- but you
sound like you have helpers for context managers that couldn't be easily
dismissed like that.


> Making such a change would involve a lot of code churn just to phase
> out a relatively obscure issue that mostly just makes certain bugs
> harder to diagnose (as was the case with the generator expression
> based izip implementation in this thread), rather than directly
> causing bugs in its own right.
>

Maybe. But the real-life version of that bug can be *really* hard to find,
and that's usually the kind of thing we'd like to fix.

FWIW the implementation of my proposal is easy to describe (which the Zen
approves of): when a StopIteration leaves a frame, replace it with some
other exception (either a new, custom one or maybe just RuntimeError),
chaining the original StopIteration.

It's the consequences that are hard to survey and describe in this case (as
they affect subtle code depending on the current semantics).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141102/9cc8bdab/attachment.html>