[Python-ideas] Possible PEP 380 tweak

[Guido van Rossum]

On Tue, Oct 26, 2010 at 7:44 AM, Jacob Holm <jh at improva.dk> wrote:
[...]
> I like this.  Having a separate function lets you explicitly request a
> return value and making it fail loudly when called on an exhausted
> generator feels just right given the prohibition against saving the
> "True" return value anywhere.  Also, using a different exception lets
> the generator distinguish between the "close" and "finish" cases, and
> making it an ordinary exception makes it clear that it is *intended* to
> be caught.  All good stuff.

I don't know. There are places where failing loudly is the right thing
to do (1 + 'a'). But when it comes to return values Python takes a
pretty strong position that there's no difference between functions
and procedures, that "return", "return None" and falling off the end
all mean the same thing, and that it's totally fine to ignore a value
or to return a value that will be of no interest for most callers.

> I am not sure that returning None when finish() cathes GeneratorReturn
> is a good idea though.  If you call finish on a generator you expect it
> to do something about it and return a value.  If the GeneratorReturn
> escapes, it is a sign that the generator was not written to expect this
> and so it likely an error.  OTOH, I am not sure it always is so maybe
> allowing it is OK.  I just don't know.
>
> How does it fit with the current PEP 380, and esp. the refactoring
> principle?   It seems like we need to special-case the GeneratorReturn
> exception somehow.  Perhaps like this:
>
> [...]
>  try:
>      _s = yield _y
> + except GeneratorReturn as _e:
> +     try:
> +         _m = _i.finish
> +     except AttributeError:
> +         raise _e  # XXX RuntimeError?
> +     raise YieldFromFinished(_m())
>  except GeneratorExit as _e:
> [...]
>
> Where YieldFromFinished inherits from GeneratorReturn, and has a 'value'
> attribute like the new StopIteration.
>
> Without something like this a function that is written to work with
> "finish" is unlikely to be refactorable.   With this, the trivial case
> of perfect delegation can be written as:
>
> def outer():
>    try:
>        return yield from inner()
>    except YieldFromFinished as e:
>        return e.value
>
> and a slightly more complex case...
>
> def outer2():
>    try:
>        a = yield from innerA()
>    except YieldFromFinished as e:
>        return e.value
>    try:
>        b = yield from innerB()
>    except YieldFromFinished as e:
>        return a+e.value
>    return a+b
>
> the "outer2" example shows why the special casing is needed.  If
> outer2.finish() is called while outer2 is suspended in innerA, a
> GeneratorReturn would be thrown directly into innerA.  Since innerA is
> supposed to be expecting this, it returns a value immediately which
> would then be the return value of the yield-from.  outer2 would then
> erroneously continue to the "b = yield from innerB()" line, which unless
> innerB immediately raised StopIteration would yield a value causing the
> outer2.finish() to raise a RuntimeError...
>
> We can avoid the extra YieldFromFinished exception if we let the new
> GeneratorReturn exception grow a value attribute instead and use it for
> both purposes.  But then the distinction between a GeneratorReturn that
> is thrown in by "finish" (which has no associated value) and the
> GeneratorReturn raised by the yield-from (which has) gets blurred a bit.
>
> Another idea is to actually replace YieldFromFinished with StopIteration
> or a GeneratorReturn inheriting from StopIteration.  That would mean we
> could drop the first try-except block in each of the above example
> generators because the "finished" result from the inner function is
> returned directly anyway.  On the other hand, that could easily lead to
> subtle bugs if you forget a try...except block that is actually needed,
> like the second block in outer2.

I'm afraid that all was too much to really reach my brain, which keeps
telling me "he's commenting on Nick's proposal which I've already
rejected".

> A different way to handle this would be to change the PEP 380 expansion
> as follows:
>
> [...]
> - except GeneratorExit as _e:
> + except (GeneratorReturn, GeneratorExit) as _e:
> [...]

That just strikes me as one more reason why a separate GeneratorReturn
is a bad idea.

In my ideal world, you almost never need to catch or raise
StopIteration; you don't raise GeneratorExit (that is close()'s job)
but you catch it to notice that your data source is finished, and then
you return a value. (And see my crazy idea in my previous post to get
rid of that too. :-)

> What this means is that only the outermost generator would see the
> GeneratorReturn.  If the outermost generator is suspended using
> yield-from, and finish() is called.  The inner generator is simply
> closed and the GeneratorReturn re-raised.  This version is only really
> useful for delegating to generators that *don't* return a value, but it
> is simpler and at least it allows *some* use of yield-from with "finish".
>
>
>> (Why "finish" as the suggested name for the method? I'd prefer
>> "return", but that's a keyword and "return_" is somewhat ugly. Pairing
>> GeneratorReturn with finish() is my second choice, for the "OK, time
>> to wrap things up and complete your assigned task" connotations, as
>> compared to the "drop everything and clean up the mess" connotations
>> of GeneratorExit and close())
>
> I like the names.  GeneratorFinish might work as well for the exception,
> but I like GeneratorReturn better for its connection with "return".
>
>
>>
>> I'd personally be +1 on option 2 (since it addresses the immediate use
>> case while maintaining appropriate separation of concerns between
>> guaranteed resource cleanup and graceful completion of coroutines) and
>> -0 on option 1 (unsurprising, given my previously stated objections to
>> failing to maintain appropriate separation of concerns).
>>
>
> I agree the "finish" idea looks far better for generators without
> yield-from.  It is unfortunate that extending it to work with yield-from
> isn't prettier that it is though.
>
>
>
>> (I should note that this differs from the previous suggestion of a
>> GeneratorReturn exception in the context of PEP 380. Those suggestions
>> were to use it as a replacement for StopIteration when a generator
>> contained a return statement. The suggestion here is to instead use it
>> as a replacement for GeneratorExit in order to request
>> prompt-but-graceful completion of a generator rather than just bailing
>> out immediately).
>
> I agree the name fits this use better than the original.  Too bad some
> of my suggestions above are starting to blur the line between
> GeneratorReturn and StopIteration again.

So now I'm even more convinced that it's not worth it...

-- 
--Guido van Rossum (python.org/~guido)