[Python-Dev] Please reconsider PEP 479.

[Nick Coghlan]

On 26 November 2014 at 18:30, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Guido van Rossum wrote:
>>
>> Hm, that sounds like you're either being contrarian or Chris and I have
>> explained it even worse than I thought.
>
> I'm not trying to be contrary, I just think the PEP could
> explain more clearly what you're trying to achieve. The
> rationale is too vague and waffly at the moment.
>
>> Currently, there are cases where list(x for x in xs if P(x)) works while
>> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the
>> PEP, both cases will raise some exception
>
> That's a better explanation, I think.

The other key aspect is that it changes the answer to the question
"How do I gracefully terminate a generator function?". The existing
behaviour has an "or" in the answer: "return from the generator frame,
OR raise StopIteration from the generator frame". That then leads to
the follow on question: "When should I use one over the other?".

The "from __future__ import generator_stop" answer drops the "or", so
it's just: "return from the generator frame".

Raising *any* exception inside the generator, including StopIteration,
then counts as non-graceful termination, bringing generators into line
with the PEP 343 philosophy that "hiding flow control in macros makes
your code inscrutable", where here, the hidden flow control is relying
on the fact that a called function raising StopIteration will
currently always gracefully terminate generator execution.

The key downside is that it means relatively idiomatic code like:

    def my_generator():
        ...
        yield next(it)
        ...

Now needs to be written out explicitly as:

    def my_generator():
        ...
       try:
            yield next(it)
        except StopIteration
            return
        ...

That's not especially easy to read, and it's also going to be very
slow when working with generator based producer/consumer pipelines.

After thinking about that concern for a while, I'd like to suggest the
idea of having a new builtin "allow_implicit_stop" decorator that
swaps out a GENERATOR code object that has the new "EXPLICIT_STOP"
flag set for one with it cleared (attempting to apply
"allow_implicit_stop" to a normal function would be an error).

Then the updated version of the above example would become:

    @allow_implicit_stop
    def my_generator():
        ...
        yield next(it)
        ...

Which would be semantically equivalent to:

    def my_generator():
       try:
           ...
            yield next(it)
            ...
        except StopIteration
            return

but *much* faster (especially if used in a producer/consumer pipeline)
since it would allow a single StopIteration instance to propagate
through the entire pipeline, rather than creating and destroying new
ones at each stage.

Including such a feature in the PEP would also make the fix to
contextlib simpler: we'd just update it so that
contextlib._GeneratorContextManager automatically calls
"allow_implicit_stop" on the passed in generator functions.

Single-source Python 2/3 code would also benefit in a 3.7+ world,
since libraries like six and python-future could just define their own
version of "allow_implicit_stop" that referred to the new builtin in
3.5+, and was implemented as an identity function in other versions.

Cheers,
Nick.

P.S. While I'm less convinced this part is a good idea, if
"allow_implicit_stop" accepted both generator functions *and*
generator objects, then folks could even still explicitly opt in to
the "or stop()" trick - and anyone reading the code would have a name
to look up to see what was going on.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia