On Mon, Oct 25, 2010 at 6:35 PM, Jacob Holm
On 2010-10-25 17:13, Guido van Rossum wrote:
On Mon, Oct 25, 2010 at 3:19 AM, Jacob Holm
wrote: More interesting (to me at least) is that this is an excellent example of why I would like to see a version of PEP380 where "close" on a generator can return a value (AFAICT the version of PEP380 on http://www.python.org/dev/peps/pep-0380 is not up-to-date and does not mention this possibility, or even link to the heated discussion we had on python-ideas around march/april 2009).
Can you dig up the link here?
I recall that discussion but I don't recall a clear conclusion coming from it -- just heated debate.
Well here is a recap of the end of the discussion about how to handle generator return values and g.close().
Thanks, very thorough!
Gregs conclusion that g.close() should not return a value: http://mail.python.org/pipermail/python-ideas/2009-April/003959.html
My reply (ordered list of ways to handle return values in generators): http://mail.python.org/pipermail/python-ideas/2009-April/003984.html
Some arguments for storing the return value on the generator: http://mail.python.org/pipermail/python-ideas/2009-April/004008.html
Some support for that idea from Nick: http://mail.python.org/pipermail/python-ideas/2009-April/004012.html
You're not convinced by Gregs argument: http://mail.python.org/pipermail/python-ideas/2009-April/003985.html
Greg arguing that using GeneratorExit this way is bad: http://mail.python.org/pipermail/python-ideas/2009-April/004001.html
You add a new complete proposal including g.close() returning a value: http://mail.python.org/pipermail/python-ideas/2009-April/003944.html
I point out some problems e.g. with the handling of return values: http://mail.python.org/pipermail/python-ideas/2009-April/003981.html
Then the discussion goes on at length about the problems of using a coroutine decorator with yield-from. At one point I am arguing for generators to keep a reference to the last value yielded: http://mail.python.org/pipermail/python-ideas/2009-April/004032.html
And you reply that storing "unnatural" state on the generator or frame object is a bad idea: http://mail.python.org/pipermail/python-ideas/2009-April/004034.html
From which I concluded that having g.close() return a value (the same on each successive call) would be a no-go: http://mail.python.org/pipermail/python-ideas/2009-April/004040.html
Which you confirmed: http://mail.python.org/pipermail/python-ideas/2009-April/004041.html
The latest draft (#13) I have been able to find was announced in http://mail.python.org/pipermail/python-ideas/2009-April/004189.html
And can be found at http://mail.python.org/pipermail/python-ideas/attachments/20090419/c7d72ba8/...
Hmm... It does look like the PEP editors dropped the ball on this one (or maybe Greg didn't mail it directly to them). It doesn't seem there are substantial differences with the published version at http://www.python.org/dev/peps/pep-0380/ though, close() still doesn't return a value.
I had some later suggestions for how to change the expansion, see e.g. http://mail.python.org/pipermail/python-ideas/2009-April/004195.html (I find that version easier to reason about even now 1½ years later)
Hopefully you & Greg can agree on a new draft. I like this to make progress and I really want this to appear in 3.3. But I don't have the time to do the editing and reviewing of the PEP.
Based on my example I have to agree that returning a value from close() would be nice. There is a little detail, how multiple arguments to StopIteration should be interpreted, but that's not so important if it's being raised by a return statement.
Right. I would assume that the return value of g.close() if we ever got one was to be taken from the first argument to the StopIteration.
That's a reasonable position. Monocle currently makes it so that using yield Return(x, y, z) [which in my view should be spelled raise Return(x, y, z0] is equivalent to return x, y, z, but there's no real need if the latter syntax is actually supported.
What killed the proposal last time was the question of what should happen when you call g.close() on an exhausted generator. My preferred solution was (and is) that the generator should save the value from the terminating StopIteration (or None if it ended by some other means) and that g.close() should return that value each time and g.next(), g.send() and g.throw() should raise a StopIteration with the value. Unless you have changed your position on storing the return value, that solution is dead in the water.
I haven't changed my position. Closing a file twice doesn't do anything the second time either.
For this use case we don't actually need to call close() on an exhausted generator so perhaps there is *some* use in only returning a value when the generator is actually running.
:-)
Here's a stupid idea... let g.close take an optional argument that it can return if the generator is already exhausted and let it return the value from the StopIteration otherwise.
def close(self, default=None): if self.gi_frame is None: return default try: self.throw(GeneratorExit) except StopIteration as e: return e.args[0] except GeneratorExit: return None else: raise RuntimeError('generator ignored GeneratorExit')
You'll have to explain why None isn't sufficient.
I totally agree that not having to call throw() and catch whatever it bounces back is much nicer. (Now I wish there was a way to avoid the "try..except GeneratorExit" construct in the generator, but I think I should stop while I'm ahead. :-)
The interesting thing is that I've been dealing with generators used as coroutines or tasks intensely on and off since July, and I haven't had a single need for any of the three patterns that this example happened to demonstrate:
- the need to "prime" the generator in a separate step - throwing and catching GeneratorExit - getting a value from close()
(I did have a lot of use for send(), throw(), and extracting a value from StopIteration.)
I think these things (at least priming and close()) are mostly an issue when using coroutines from non-coroutines. That means it is likely to be common in small examples where you write the whole program, but less common when you are writing small(ish) parts of a larger framework.
Throwing and catching GeneratorExit is not common, and according to some shouldn't be used for this purpose at all.
Well, *throwing* it is close()'s job. And *catching* it ought to be pretty rare. Maybe this idiom would be better: def sum(): total = 0 try: while True: value = yield total += value finally: return total
In my context, generators are used to emulate concurrently running tasks, and "yield" is always used to mean "block until this piece of async I/O is complete, and wake me up with the result". This is similar to the "classic" trampoline code found in PEP 342.
In fact, when I wrote the example for this thread, I fumbled a bit because the use of generators there is different than I had been using them (though it was no doubt thanks to having worked with them intensely that I came up with the example quickly).
This sounds a lot like working in a "larger framework" to me. :)
Possibly. I realize that I have code something like this: next_input = None while ...not done yet...: output = gen.send(next_input) next_input = ...computed from output... # many variations which quite naturally computes next_input from output but it does start out with an initial value of None for next_input in order to prime the pump.
So, it is clear that generators are extremely versatile, and PEP 380 deserves several good use cases to explain all the API subtleties.
I like your example because it matches the way I would have used generators to solve it. OTOH, it is not hard to rewrite parallel_reduce as a traditional function. In fact, the result is a bit shorter and quite a bit faster so it is not a good example of what you need generators for.
I'm not sure I understand. Maybe you meant to rewrite it as a class? There's some state that wouldn't have a good place to live without either a class or a (generator) stackframe to survive.
BTW, while I have you, what do you think of Greg's "cofunctions" proposal?
I'll have to get back to you on that.
- Jacob
-- --Guido van Rossum (python.org/~guido)