[Python-ideas] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Sun, Nov 16, 2014 at 3:51 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 November 2014 01:56, Chris Angelico <rosuav at gmail.com> wrote:
>> Okay, let me see if I have this straight. When a 'return' statement
>> (including an implicit one at end-of-function) is encountered in any
>> function which contains a 'yield' statement, it is implemented as
>> "raise GeneratorReturn(value)" rather than as "raise
>> StopIteration(value)" which is the current behaviour. However, if any
>> GeneratorReturn would be raised in any way other than the 'return'
>> statement, it would magically become a StopIteration instead. Is that
>> correct?
>
> When you call next(gi), you're invoking the __next__ method on the
> *generator iterator*. It's that method which restarts evaluation of the
> generator frame at the point where it last left off, and interprets any
> results.
>
> Now, there are three things that can happen as a result of that frame
> evaluation:
>
>     1. It hits a yield point. In that case, gi.__next__ returns the yielded
> value.
>
>     2. It can return from the frame. In that case. gi.__next__ creates a
> *new* StopIteration instance (with an appropriate return value set) and
> raises it
>
>     3. It can throw an exception. In that case, gi.__next__ just allows it
> to propagate out (including if it's StopIteration)

Thank you for explaining.
-- Cameron

In case others were also oversimplifying in their heads, I've
summarized the above into the PEP.

> (The possible outcomes of gi.send() and gi.throw() are the same as those of
> next(gi). gi.throw() has the novel variant where the exception thrown in may
> propagate back out)

Should that variant affect this proposal? What should happen if you
throw StopIteration or GeneratorReturn into a generator?

> The two change proposals being discussed are as follows:
>
> Current PEP (backwards incompatible): Change outcome 3 to convert
> StopIteration to RuntimeError (or a new exception type). Nothing else
> changes.
>
> Alternative (backwards compatible): Change outcome 2 to raise
> GeneratorReturn instead of StopIteration and outcome 3 to convert
> GeneratorReturn to StopIteration.
>
> The alternative *doesn't* do anything about the odd discrepancy between
> comprehensions and generator expressions that started the previous thread.
> It just adds a new capability where code that knows it's specifically
> dealing with a generator (like contextlib or asyncio) can more easily tell
> the difference between outcomes 2 and 3.

Text along these lines added to PEP, thanks!

>> Question: How does it "become" StopIteration? Is a new instance of
>> StopIteration formed which copies in the other's ``value``? Is the
>> type of this exception magically altered? Or is it a brand new
>> exception with the __cause__ or __context__ set to carry the original?
>
> I'd suggest used the exception chaining machinery and creating a new
> exception with __cause__ and the generator return value set appropriately.

Makes sense. If the __cause__ is noticed at all (ie this doesn't just
quietly stop a loop), it wants to be very noisy.

>> Thanks, I figured it'd be like that. Since contextlib exists in 2.7,
>> is contextlib2 meant to be legacy support only?
>
> contextlib has actually been around since 2.5, but some features (most
> notably ExitStack) weren't added until much later. Like unittest2,
> contextlib2 allows access to newer stdlib features on older versions (I
> haven't used it as a testing ground for new ideas since ExitStack).

If there is breakage from this, it would simply mean "older versions
of contextlib2 are not compatible with Python 3.7, please upgrade your
contextlib2" - several of the variants make it perfectly possible to
write cross-version-compatible code. I would hope that this remains
the case.

ChrisA