[Python-Dev] PEP 479: Change StopIteration handling inside generators

[Raymond Hettinger]

> On Nov 22, 2014, at 6:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> I'm definitely coming around to the point of view that, even if we wouldn't design it the way it currently works given a blank slate, the alternative design doesn't provide sufficient benefit to justify the cost of changing the behaviour & getting people to retrain their brains.

Thanks Nick.  That was well said.

After a couple more days of thinking about PEP 455 and reading many
of the mailing list posts, I am also still flatly opposed to the proposal
and wanted to summarize my thoughts for everyone's consideration.

It looks like the only point in favor of the PEP is makes the internal
semantics of genexps appear to more closely match list comprehensions.

However, it does so not by fixing anything or adding a capability;
rather, it disallows a coding pattern that potentially exposes the
true differences between genexps and list comps.  Even then, the
difference isn't hidden; instead, the proposal just breaks the code
loudly by raising a RuntimeError.

AFAICT, the problem it solves isn't really a problem in practice.
(I do code reviews and teach Python for living, so I get broad
exposure to how python is used in practice).

As collateral damage, the PEP breaks code that is otherwise
well designed, beautiful looking, and functioning correctly.

Even worse, the damage will be long lasting.  In introduces
a new special case in the current clean design of generators.
And, it creates an additional learning burden (we would now
have to teach the special case and how to work around it
with a "try: v=next(it)  except StopIteration: return".

I realize these are sweeping statements, so I elaborate with more
detail and examples below.  If you're not interested in the details,
feel free to skip the rest of the post; you've already gotten the keys
points.


New Special Case
----------------

By design, exceptions raised in generators are passed through to the
caller.  This includes IndexErrors, ValueErrors, StopIteration, and
PEP 342's GeneratorExit.  Under the proposed PEP, there general
rule (exceptions are passed through) is broken and there is a new
special case:  StopIteration exceptions are caught and reraised
as RuntimeErrors.  This is a surprising new behavior.


Breaks well established, documented, and tested behaviors
---------------------------------------------------------

From the first day generators were introduced 13 years ago, we have
documented and promised that you can end a terminate a generator by
raising StopIteration or by a return-statment (that is in PEP 255,
in the whatsnew document for 2.2, in the examples we provided for the
myriad of ways to use generators, in standard library code, in the
Martelli's Python Cookbook example, in the documention for itertools,
etc.)


Legitimate Use Cases for Raising StopIteration in a Generator
------------------------------------------------------------

In a producer/consumer generator chain, the input generator signals
it is done by raising StopIteration and the output generator signals
that it is done by raising StopIteration (this isn't in dispute).

That use case is currently implemented something like this:

    def middleware_generator(source_generator):
        it = source_generator()
        input_value = next(it)
        output_value = do_something_interesting(input_value)
        yield output_value

Termination of the input stream will then terminate middleware stream.
You can see several real world examples of this code pattern in
Fredrik Lundh's pure python verion of ElementTree
(prepare_descendant, prepare_predicate, and iterfind).

Under the proposal, the new logic would be to:
1) catch input stream StopIteration
2) return from the generator
3) which in turn raises StopIteration yet again.

This doesn't make the code better in any way.  The new code
is wordy, slow, and unnecessarily convoluted:

    def middleware_generator(source_generator):
        it = source_generator()
        try:
            input_value = next(it)
        except StopIteration:
            return             # This causes StopIteration to be reraised
        output_value = do_something_interesting(input_value)
        yield output_value

I don't look forward to teaching people to have to write code like this
and to have to remember yet another special case rule for Python.


Is next() Surprising?
---------------------

The PEP author uses the word "surprise" many times in describing the
motivation for the PEP.  In the context of comparing generator expressions
to list comprehenions, I can see where someone might be surprised that
though similar in appearance, their implementations are quite different
and that some of those differences might not expected.

However, I believe this is where the "surprise" ends.

The behavior of next(it) is to return a value or raise StopIteration.
That is fundamental to what is does (what else could it do?).

This is as basic as list indexing returning a value or raising an IndexError,
or as dict lookups returning a value or raising a KeyError.

If someone in this thread says that they were suprised that next() could
raise StopIteration, I don't buy it.

Being able to consume a value from an iterator stream is a fundamental
skill and not hard to learn (when I teach iterators and generators, the
operation of next() has never been a stumbling block).

The Python iterator protocol isn't rocket science.  Like almost every
other language, it is based on the Iterator Design Pattern found in
the Gang of Four Design patterns book.

For the discussion of the PEP to go forward, it is necessary to stop
"acting surprised" about everything related to iterators and
generators.  If there is any surprise, it is confined to uncommon cases
that make visible the differences between generator expressions and
list comprehensions (IIRC, that is how this thread started).

Also, it pays to remember that we have 13 years worth of real world
experience with generators and iterators in Python.  By and large,
it has been one of our biggest success stories.   Please keep that
in mind when experiencing the temptation to change it.

TL;DR

Sorry for the long post, but I've been thinking about this deeply for
a couple days and wanted to get all the thoughts out to others for
consideration,


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141122/7c3d87f6/attachment.html>