[Python-ideas] Generators are iterators

Oscar Benjamin oscar.j.benjamin at gmail.com
Fri Dec 12 14:58:22 CET 2014

On 12 December 2014 at 13:14, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 12 December 2014 at 22:42, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>> On 12 December 2014 at 10:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> The
>>> specific problem deemed worthy of being fixed is that the presence of
>>> "yield" in a function body can implicitly suppress StopIteration
>>> exceptions raised elsewhere in that function body (or in functions it
>>> calls).
>> The yield causes the function to become a generator function. The
>> frame for a generator function (like for any other function) will
>> allow uncaught exceptions to propagate to the frame above. The
>> difference between generator functions and other functions is that the
>> code in the body of a generator function is executed when someone
>> calls the generator's __next__ method. Since the caller (the iterator
>> consumer) is expecting StopIteration it is treated as the signalling
>> the end of iteration. The yield suppresses nothing; it is the iterator
>> consumer e.g. the for-loop or the list() function etc. which catches
>> the StopIteration and treats it as termination.
> Oscar, this is where you're *just plain wrong* from a technical perspective.

Which of the above sentences is technically incorrect?

> The conversion to a generator function also introduces an additional
> layer of indirection at iteration time through the
> generator-iterator's __next__ method implementation.
> That indirection layer is the one that converts an ordinary return
> from the generator frame into a StopIteration exception. Historically,
> it also allowed StopIteration from within the frame to propagate out,
> make it impossible to tell from outside the generator iterator whether
> the termination was due to the generator returning normally, or due to
> StopIteraton being raised.

I understand all of this.

> The introduction of "yield from" in Python 3.3, and its use in the
> coroutine design for the asyncio library and similar contexts *makes
> that distinction more important than it used to be*. It even managed
> to introduce obscure bugs around the use of generators to write
> context managers, where factoring out part of the context manager to a
> subgenerator may implicitly suppress a StopIteration exception raised
> in the body of the with statement.
> The problem you think PEP 479 is trying to solve *is* the one where
> the discussion started, but it is *not* the one where it ended when
> Guido accepted PEP 479. The problems PEP 479 solves are generator
> specific - they have nothing to do with the iterator protocol in
> general.

Okay well this is not clear from the PEP (and bear in mind that I did
take the time to read through all of the lengthy threads on this
subject before posting here). I haven't read the most recent changes
but when I did read the PEP the justification appeared to stem from
claims that were confusing at best and plain false at worst e.g.
"generators are not iterators".


More information about the Python-ideas mailing list