[Python-ideas] Possible PEP 380 tweak

[Guido van Rossum]

On Mon, Oct 25, 2010 at 6:35 PM, Jacob Holm <jh at improva.dk> wrote:
> On 2010-10-25 17:13, Guido van Rossum wrote:
>> On Mon, Oct 25, 2010 at 3:19 AM, Jacob Holm <jh at improva.dk> wrote:
>>> More interesting (to me at least) is that this is an excellent example
>>> of why I would like to see a version of PEP380 where "close" on a
>>> generator can return a value (AFAICT the version of PEP380 on
>>> http://www.python.org/dev/peps/pep-0380 is not up-to-date and does not
>>> mention this possibility, or even link to the heated discussion we had
>>> on python-ideas around march/april 2009).
>>
>> Can you dig up the link here?
>>
>> I recall that discussion but I don't recall a clear conclusion coming
>> from it -- just heated debate.
>>
>
>
> Well here is a recap of the end of the discussion about how to handle
> generator return values and g.close().

Thanks, very thorough!

>  Gregs conclusion that g.close() should not return a value:
>  http://mail.python.org/pipermail/python-ideas/2009-April/003959.html
>
>  My reply (ordered list of ways to handle return values in generators):
>  http://mail.python.org/pipermail/python-ideas/2009-April/003984.html
>
>  Some arguments for storing the return value on the generator:
>  http://mail.python.org/pipermail/python-ideas/2009-April/004008.html
>
>  Some support for that idea from Nick:
>  http://mail.python.org/pipermail/python-ideas/2009-April/004012.html
>
>  You're not convinced by Gregs argument:
>  http://mail.python.org/pipermail/python-ideas/2009-April/003985.html
>
>  Greg arguing that using GeneratorExit this way is bad:
>  http://mail.python.org/pipermail/python-ideas/2009-April/004001.html
>
>  You add a new complete proposal including g.close() returning a value:
>  http://mail.python.org/pipermail/python-ideas/2009-April/003944.html
>
>  I point out some problems e.g. with the handling of return values:
>  http://mail.python.org/pipermail/python-ideas/2009-April/003981.html
>
>  Then the discussion goes on at length about the problems of using a
>  coroutine decorator with yield-from.  At one point I am arguing for
>  generators to keep a reference to the last value yielded:
>  http://mail.python.org/pipermail/python-ideas/2009-April/004032.html
>
>  And you reply that storing "unnatural" state on the generator or
>  frame object is a bad idea:
>  http://mail.python.org/pipermail/python-ideas/2009-April/004034.html
>
>  From which I concluded that having g.close() return a value (the same
>  on each successive call) would be a no-go:
>  http://mail.python.org/pipermail/python-ideas/2009-April/004040.html
>
>  Which you confirmed:
>  http://mail.python.org/pipermail/python-ideas/2009-April/004041.html
>
>
> The latest draft (#13) I have been able to find was announced in
> http://mail.python.org/pipermail/python-ideas/2009-April/004189.html
>
> And can be found at
> http://mail.python.org/pipermail/python-ideas/attachments/20090419/c7d72ba8/attachment-0001.txt

Hmm... It does look like the PEP editors dropped the ball on this one
(or maybe Greg didn't mail it directly to them). It doesn't seem there
are substantial differences with the published version at
http://www.python.org/dev/peps/pep-0380/ though, close() still doesn't
return a value.

> I had some later suggestions for how to change the expansion, see e.g.
> http://mail.python.org/pipermail/python-ideas/2009-April/004195.html  (I
> find that version easier to reason about even now 1½ years later)

Hopefully you & Greg can agree on a new draft. I like this to make
progress and I really want this to appear in 3.3. But I don't have the
time to do the editing and reviewing of the PEP.

>> Based on my example I have to agree that returning a value from
>> close() would be nice. There is a little detail, how multiple
>> arguments to StopIteration should be interpreted, but that's not so
>> important if it's being raised by a return statement.
>>
>
> Right.  I would assume that the return value of g.close() if we ever got
> one was to be taken from the first argument to the StopIteration.

That's a reasonable position. Monocle currently makes it so that using
yield Return(x, y, z) [which in my view should be spelled raise
Return(x, y, z0] is equivalent to return x, y, z, but there's no real
need if the latter syntax is actually supported.

> What killed the proposal last time was the question of what should
> happen when you call g.close() on an exhausted generator.  My preferred
> solution was (and is) that the generator should save the value from the
> terminating StopIteration (or None if it ended by some other means) and
> that g.close() should return that value each time and g.next(), g.send()
> and g.throw() should raise a StopIteration with the value.
> Unless you have changed your position on storing the return value, that
> solution is dead in the water.

I haven't changed my position. Closing a file twice doesn't do
anything the second time either.

> For this use case we don't actually need to call close() on an exhausted
> generator so perhaps there is *some* use in only returning a value when
> the generator is actually running.

:-)

> Here's a stupid idea... let g.close take an optional argument that it
> can return if the generator is already exhausted and let it return the
> value from the StopIteration otherwise.
>
> def close(self, default=None):
>    if self.gi_frame is None:
>        return default
>    try:
>        self.throw(GeneratorExit)
>    except StopIteration as e:
>        return e.args[0]
>    except GeneratorExit:
>        return None
>    else:
>        raise RuntimeError('generator ignored GeneratorExit')

You'll have to explain why None isn't sufficient.

>> I totally agree that not having to call throw() and catch whatever it
>> bounces back is much nicer. (Now I wish there was a way to avoid the
>> "try..except GeneratorExit" construct in the generator, but I think I
>> should stop while I'm ahead. :-)
>>
>> The interesting thing is that I've been dealing with generators used
>> as coroutines or tasks intensely on and off since July, and I haven't
>> had a single need for any of the three patterns that this example
>> happened to demonstrate:
>>
>> - the need to "prime" the generator in a separate step
>> - throwing and catching GeneratorExit
>> - getting a value from close()
>>
>> (I did have a lot of use for send(), throw(), and extracting a value
>> from StopIteration.)
>>
>
> I think these things (at least priming and close()) are mostly an issue
> when using coroutines from non-coroutines.  That means it is likely to
> be common in small examples where you write the whole program, but less
> common when you are writing small(ish) parts of a larger framework.
>
> Throwing and catching GeneratorExit is not common, and according to some
> shouldn't be used for this purpose at all.

Well, *throwing* it is close()'s job. And *catching* it ought to be
pretty rare. Maybe this idiom would be better:

def sum():
  total = 0
  try:
    while True:
      value = yield
      total += value
  finally:
    return total

>> In my context, generators are used to emulate concurrently running
>> tasks, and "yield" is always used to mean "block until this piece of
>> async I/O is complete, and wake me up with the result". This is
>> similar to the "classic" trampoline code found in PEP 342.
>>
>> In fact, when I wrote the example for this thread, I fumbled a bit
>> because the use of generators there is different than I had been using
>> them (though it was no doubt thanks to having worked with them
>> intensely that I came up with the example quickly).
>>
>
> This sounds a lot like working in a "larger framework" to me. :)

Possibly. I realize that I have code something like this:

next_input = None
while ...not done yet...:
  output = gen.send(next_input)
  next_input = ...computed from output...  # many variations

which quite naturally computes next_input from output but it does
start out with an initial value of None for next_input in order to
prime the pump.

>> So, it is clear that generators are extremely versatile, and PEP 380
>> deserves several good use cases to explain all the API subtleties.
>>
>
> I like your example because it matches the way I would have used
> generators to solve it.  OTOH, it is not hard to rewrite parallel_reduce
> as a traditional function.  In fact, the result is a bit shorter and
> quite a bit faster so it is not a good example of what you need
> generators for.

I'm not sure I understand. Maybe you meant to rewrite it as a class?
There's some state that wouldn't have a good place to live without
either a class or a (generator) stackframe to survive.

>> BTW, while I have you, what do you think of Greg's "cofunctions" proposal?
>>
>
> I'll have to get back to you on that.
>
> - Jacob
>



-- 
--Guido van Rossum (python.org/~guido)