[Python-ideas] Generators are iterators

[Nick Coghlan]

On 13 December 2014 at 09:15, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
> I'm just saying that it's a mistake that can easily be made. Since I
> made it I'm now super-suspicious of next() so I probably wouldn't make
> it again. This mistake is the root of all of the problems that PEP 479
> attempts to solve. If you don't believe it is a problem that someone
> might make this mistake then it follows that PEP 479 is a massive
> mistake.

Oscar, StopIteration unexpectedly escaping from __next__ methods is
*not* the problem PEP 479 is trying to solve, and that
misunderstanding appears to be the fundamental error causing your
concern. PEP 479 is aiming to solve a specific problem with *writing
generator functions*, not the more general problem with invoking
iterators that bothers you. There is nothing to be done about
StopIteration escaping from __next__ methods in general - it's an
inherent limitation of the iterator protocol. StopIteration escaping
from generator frames is a different story, as the generator-iterator
__next__ method implementation has the chance to intercept them and
convert them to something else.

So let me try again from the pedagogical angle in a post PEP 380 Python world.

If you are teaching someone to write explicit __next__ methods, here
are the things you *absolutely* need to teach them:

1. Values returned from the method are produced as values in the iterator
2. Raising StopIteration terminates the iterator
3. If invoked via "yield from", the argument to StopIteration is the
result of the "yield from" expression (default: None)
4. If delegating to a function via a function call, points 2 & 3 also
apply to that subfunction (this includes the builtin next() function)

If you are instead teaching someone to write generator functions, here
are the things you *absolutely* need to teach them:

1. Values passed to yield expressions are produced as values in the iterator
2. Returning from the frame terminates the iterator
3. If invoked via "yield from", the return value of the generator
frame is the result of the "yield from" expression (default: None)
4. If delegating to a subgenerator (rather than to an arbitrary
iterator) via a "yield from" expression, then next(), send() and
throw() are passed to that subgenerator until it terminates, at which
point execution resumes in the parent generator frame (either
producing a value from the yield from expression, or raising an
exception if the subgenerator failed)

With just those 8 points (and their existing knowledge regarding how
to *use* iterators and generators), a developer is now equipped to
write full featured __next__ method and generator function
implementations, including delegation to subiterators and
subgenerators via "yield from" in the latter case.

Unfortunately, in both Python 3.3 and 3.4, there's a problem with
these instructions: they leave something out. The thing they leave out
is the fact that points 2-4 from the section on __next__ methods
*also* apply to writing generator functions. Even though it's
completely unnecessary, and usually not the right approach if only
supporting modern versions of Python, new users still need to be
taught it, because it can have unexpected side effects on the way
their generators work.

This redundancy increases the cognitive complexity of generator
functions: new users need to be taught *two* ways of doing something,
and then told "don't use the second way, we're only teaching it to you
because someone else might use it in code you end up reading or
maintaining, or because you might hit it accidentally and be
struggling to figure out why your generator based iteration is
unexpectedly finishing early".

*That's* the problem PEP 479 solves: it's takes those 3 bullet points
regarding the behaviour of StopIteration inside __next__ method
implementations and makes them *no longer apply* to writing generator
functions. If folks do run into them, they'll get a RuntimError and
have the opportunity to debug it, and figure out where that came from.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia