[Python-Dev] PEP 479: Change StopIteration handling inside generators

[Raymond Hettinger]

> On Nov 19, 2014, at 12:10 PM, Guido van Rossum <guido at python.org> wrote:
> 
> There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate.
> 
> The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools.

I strongly recommend against accepting this PEP.

The PEP itself eloquently articulates an important criticism, "Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]  <https://www.python.org/dev/peps/pep-0479/#id16>Code does exist which relies on the current behaviour (e.g. [2]  <https://www.python.org/dev/peps/pep-0479/#id14>, [5]  <https://www.python.org/dev/peps/pep-0479/#id17>, [6]  <https://www.python.org/dev/peps/pep-0479/#id18>), and there is the concern that this would be unnecessary code churn to achieve little or no gain.".

Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators.  However, Chris explained that, "This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing."  That is a major and worrisome conceptual shift.

Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) inside a generator and letting the generator terminate when the data stream  ends.  Here is an example that I have taught for years:

    def izip(iterable1, iterable2):
        it1 = iter(iterable1)
        it2 = iter(iterable2)
        while True:
            v1 = next(it1)
            v2 = next(it2)
            yield v1, v2

The above code is not atypical.  Several of the pure python equivalents in the itertools docs have documented this pattern to the world for over a decade.  I have seen it other people's code bases as well (in several contexts including producer/consumer chains, generators that use next() to fetch initial values from a stream, and generators that have multiple subiterators).  This behavior was guaranteed from day one in PEP 255, so we would be breaking a long-standing, published rule.

Adding a try/except to catch the StopIteration make the above code compliant with the new PEP, but it wouldn't make the code better in any way.  And after that fix, the code would be less beautiful that it is now, and I think that matters.

Lastly, as I mentioned on python-ideas, if we really want people to migrate to Python 3, there should be a strong aversion to further increasing the semantic difference between Python 2 and Python 3 without a really good reason.


Raymond


P.S.  The PEP 255 promise was also announced as a practice in the WhatsNew 2.2 document (where most people first learned what generators are and how to use them), "The end of the generator’s results can also be indicated by raising StopIteration <https://docs.python.org/3/library/exceptions.html#StopIteration> manually, or by just letting the flow of execution fall off the bottom of the function."   The technique was also used in the generator tutorial (which was tested Lib/test/test_generators.py).

One other thought:  A number of other languages have added generators modeled on the Python implementation.   It would be worthwhile to check to see what the prevailing wisdom is regarding whether it should be illegal to raise StopIteration inside a generator.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141121/d4b77c5a/attachment.html>