[Python-ideas] Generators are iterators

[Nick Coghlan]

On 12 December 2014 at 22:42, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
> On 12 December 2014 at 10:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 12 December 2014 at 01:14, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>>>
>>> I think the PEP would be clearer if it properly acknowledged that the
>>> problem is a problem for all iterators. The question then is why the
>>> fix is only targeted at generators and what should be done about the
>>> same problem that occurs in many other forms. The PEP rationale avoids
>>> these issues by falsely claiming that generators are special.
>>
>> I believe you're misunderstanding the problem being solved.
>
> And I believe you are too. :)
>
>> The
>> specific problem deemed worthy of being fixed is that the presence of
>> "yield" in a function body can implicitly suppress StopIteration
>> exceptions raised elsewhere in that function body (or in functions it
>> calls).
>
> The yield causes the function to become a generator function. The
> frame for a generator function (like for any other function) will
> allow uncaught exceptions to propagate to the frame above. The
> difference between generator functions and other functions is that the
> code in the body of a generator function is executed when someone
> calls the generator's __next__ method. Since the caller (the iterator
> consumer) is expecting StopIteration it is treated as the signalling
> the end of iteration. The yield suppresses nothing; it is the iterator
> consumer e.g. the for-loop or the list() function etc. which catches
> the StopIteration and treats it as termination.

Oscar, this is where you're *just plain wrong* from a technical perspective.

The conversion to a generator function also introduces an additional
layer of indirection at iteration time through the
generator-iterator's __next__ method implementation.

That indirection layer is the one that converts an ordinary return
from the generator frame into a StopIteration exception. Historically,
it also allowed StopIteration from within the frame to propagate out,
make it impossible to tell from outside the generator iterator whether
the termination was due to the generator returning normally, or due to
StopIteraton being raised.

The introduction of "yield from" in Python 3.3, and its use in the
coroutine design for the asyncio library and similar contexts *makes
that distinction more important than it used to be*. It even managed
to introduce obscure bugs around the use of generators to write
context managers, where factoring out part of the context manager to a
subgenerator may implicitly suppress a StopIteration exception raised
in the body of the with statement.

The problem you think PEP 479 is trying to solve *is* the one where
the discussion started, but it is *not* the one where it ended when
Guido accepted PEP 479. The problems PEP 479 solves are generator
specific - they have nothing to do with the iterator protocol in
general.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia