[Python-ideas] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Thu, Nov 20, 2014 at 11:25 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> I'm not sure that many people outside of this and the python-dev mailing
> lists would find the use of "raise StopIteration" surprising. Rather, I
> expect that they will find the use of an explicit "return" inside a
> generator surprising. People are informally taught that generators use
> yield *instead of* return, so seeing both in the same function is a
> surprise. (Most generators quitely fall out the bottom with no explicit
> end.)

Interesting. But "yield instead of return" doesn't automatically say
"and then raise StopIteration to early-abort"; I'd say the informal
description is fine, it just needs to be modified differently once
people actually want an early abort. ("You can still use 'return' for
its other purpose, terminating a function before reaching the end.")

> Examples of code in the wild using StopIteration to exit:
>
> http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0

Trivially unnecessary, and as soon as there's a bug report, the
"What's New In 3.7" page will explain that it needs to be removed.

> http://code.openhub.net/file?fid=WzkucGktJhjsP8cj4BO6Wcnbx-0&cid=fsj7E8vdVMA&s=raise%20StopIteration&fp=401086&mp&projSelected=true#L0

I have no idea what this one is doing, but it looks like it's half way
to what's wanted here. Catch the exception and deal with it... this
proposal just means the "deal with it" part needs to be reworded into
a return statement.

All it needs is for "What's New in 3.5" to recommend use of 'return'
instead of 'raise StopIteration', and all these cases will be easily
compatible with all [1] versions of Python.

> http://stackoverflow.com/questions/6784934/python-yield-and-stopiteration-in-one-loop

The accepted answer correctly advises the function be written to
simply return. This will work fine. The other answer has a local
"raise StopIteration", which can be translated into a simple "return".

> http://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-the-difference-between-raise-stopiteration-and
>
> That last example not only uses raise to exit the generator, but the
> author actually guesses that it is the more Pythonic way to do so.

The question's author does, but the accepted answer recommends "return".

This may result in the odd question here or there, but it's not a
major problem. Any time a generator has "raise StopIteration" in its
own body, it can simply become "return". That's easy. The issue comes
up when it's not raising that itself, but is letting it bubble up -
maybe from a next() call.

def takewhiletrue(iter):
    while True: # coincidental with the function name
        # try:
        val = next(iter)
        # except StopIteration: return
        if not val: break
        yield val

This won't come up in a simple search for "raise StopIteration", and
if you have something like this where the condition is almost always
going to be reached eventually, you might not notice the problem for a
long time. How would you know to add the commented-out lines? What
kind of code search would you use to detect this?

> Here is a description of the generator protocol which could easily lead
> the reader to conclude that raising StopIteration is the correct way to
> exit a generator:
>
>     To support this protocol, functions with yield statement
>     are compiled specially as generators. They return a generator
>     object when they are called. The returned object supports the
>     iteration interface with an automatically created __next__()
>     method to resume execution. Generator functions may have a
>     return simply terminates the generation of values by raising
>     a StopIteration exceptions after any normal function exit.
>
> http://www.bogotobogo.com/python/python_generators.php

Even that does recommend 'return'. If anyone reads that, writes "raise
StopIteration", sees code bombing with RuntimeError, and then comes to
python-list, we can explain that the recommended method is "return". I
have no problem with this. There are plenty of much-worse practices
that people pick up - mixing bytes and text, using backslashes in
Windows path names without doubling them or using raw literals, etc,
etc, etc. In lots of cases they'll seem to work ("C:\Program Files\New
Stuff\Testing" will work, until you lower-case the name), but when
they break, you just have to fix them. This wouldn't be the first
Python minor version to tighten up requirements to remove bug magnets.

> At this point, I'm convinced that there is a good argument for a
> __future__ import changing this behaviour. But I suspect that making
> this the default behaviour in the future will break a lot of code.

I suspect that a huge percentage of the code so broken can be
trivially fixed just by search/replacing "raise StopIteration" with
"return". There'll be only a very few where the StopIteration is
raised from some other function and needs to be caught and dealt with
- and fixing those is just as likely to reveal bugs needing fixing.

ChrisA