[Python-ideas] Generators are iterators

[Oscar Benjamin]

Somehow during the discussion of PEP 479 it was concluded that
generators will become distinct from iterators. A few people have
pointed out that this is a harmful distinction (complicates the
language hard to explain etc). I think it is simply an invalid
distinction and should be abolished.

PEP 479 proposes to change the way that generators handle
StopIteration fall-through by default. However the iterator protocol
has never formally specified that an iterator should allow
StopIteration to fall through to a grandparent consumer. I considered
the fact that StopIteration fall-through was possible to be a
convenient design feature but it was never a required part of the
definition of an iterator.

Many iterators perform other kinds of exception handling without
allowing that to fall through and many iterators catch StopIteration
from child iterators without reraising it. This has never lead anyone
to suggest that such iterators are not true iterators in any way.

Under PEP 479 generators will handle a StopIteration arising from the
executing frame differently. The generator will still be an object
with __iter__ and __next__ that exposes StopIteration at the
appropriate time to its parent iterator-consumer. In other words it
will still satisfy the definition of an iterator.

The PEP says:

"""
Under this proposal, generators and iterators would be distinct, but
related, concepts. Like the mixing of text and bytes in Python 2, the
mixing of generators and iterators has resulted in certain perceived
conveniences, but proper separation will make bugs more visible.
""""

This is just plain false. Under the proposal generators will still be
iterators. Mixing generators and iterators is nothing like mixing text
and bytes and never has been. Mixing iterators and iterables is a bit
like mixing text and the bytes in the sense that it can seem to work
but then sometimes silently do the wrong thing. Mixing generators and
iterators is like mixing sets and containers: there is no mixing since
it is simply a false distinction.

AFAICT from reading through the discussions this idea has (implicitly)
followed from the following fallacious argument:

1) Bare next creates a problem for generators.
2) Therefore we fix it by changing generators.
3) Therefore generators are not iterators any more.

Anyone with experience in propositional logic can see multiple
fallacies in that argument but actually the biggest mistake is simply
in the opening premise: bare next is a problem for all iterators.
Generators are affected precisely because they are iterators.

Generators were introduced as "a kind of Python iterator, but of an
especially powerful kind" (PEP 255). The coroutine idea has since
turned generators into something of a Frankenstein concept but the
fundamental fact that they are iterators remains unchanged. And
regardless of how much discussion coroutines generate the fact remains
that 99% of generators are used purely for iteration.

I propose to abolish this notion that generators are not iterators and
to amend the text of the PEP to unambiguously state that generators
are iterators regardless of any changes to the way they propagate
StopIteration from the executing frame.


Oscar