Re: [Python-ideas] Possible PEP 380 tweak
I've been pondering the whole close()-returning-a-value thing I've convinced myself once again that it's a bad idea. Essentially the problem is that we're trying to make the close() method, and consequently GeneratorExit, serve two different and incompatible roles. One role (the one it currently serves) is as an emergency bail-out mechanism. In that role, when we have a stack of generators delegating via yield-from, we want things to behave as thought the GeneratorExit originates in the innermost one and propagates back out of the entire stack. We don't want any of the intermediate generators to catch it and turn it into a StopIteration, because that would give the next outer one the misleading impression that it's business as usual, but it's not. This is why PEP 380 currently specifies that, after calling the close() method of the subgenerator, GeneratorExit is unconditionally re-raised in the delegating generator. The proponents of close()-returning-a-value, however, want GeneratorExit to serve another role: as a way of signalling to a consuming generator (i.e. one that is having values passed into it using send()) that there are no more values left to pass in. It seems to me that this is analogous to a function reading values from a file, or getting them from an iterator. The behaviour that's usually required in the presence of delegation is quite different in those cases. Consider a function f1, that calls another function f2, which loops reading from a file. When f2 reaches the end of the file, this is a signal that it should finish what it's doing and return a value to f1, which then continues in its usual way. Similarly, if f2 uses a for-loop to iterate over something, when the iterator is exhausted, f2 continues and returns normally. I don't see how GeneratorExit can be made to fulfil this role, i.e. as a "producer exhausted" signal, without compromising its existing one. And if that idea is dropped, the idea of close() returning a value no longer has much motivation that I can see. So how should "producer exhausted" be signalled, and how should the result of a consumer generator be returned? As for returning the result, I think it should be done using the existing PEP 380 mechanism, i.e. the generator executes a "return", consequently raising StopIteration with the value. A delegating generator will then see this as the result of a yield-from and continue normally. As for the signalling mechanism, I think that's entirely a matter for the producer and consumer to decide between themselves. One way would be to send() in a sentinel value, if there is a suitable out-of-band value available. Another would be to throw() in some pre-arranged exception, perhaps EOFError as a suggested convention. If we look at files as an analogy, we see a similar range of conventions. Most file reading operations return an empty string or bytes object on EOF. Some, such as readline(), raise an exception, because the empty element of the relevant type is also a valid return value. As an example, a consumer generator using None as a sentinel value might look like this: def summer(): tot = 0 while 1: x = yield if x is None: break tot += x return tot and a producer using it: s = summer() s.next() for x in values: s.send(x) try: s.send(None) except StopIteration as e: result = e.value Having to catch StopIteration is a little tedious, but it could easily be encapsulated in a helper function: def close_consumer(g, sentinel): try: g.send(sentinel) except StopIteration as e: return e.value The helper function could also take care of another issue that arises. What happens if a delegating consumer carries on after a subconsumer has finished and yields again? The analogous situation with files is trying to read from a file that has already signalled EOF before. In that case, the file simply signals EOF again. Similarly, calling next() on an exhausted iterator raises StopIteration again. So, if a "finished" consumer yields again, and we are using a sentinel value, the yield should return the sentinel again. We can get this behaviour by writing our helper function like this: def close_consumer(g, sentinel): while 1: try: g.send(sentinel) except StopIteration as e: return e.value So in summary, I think PEP 380 and current generator semantics are fine as they stand with regard to the behaviour of close(). Signalling the end of a stream of values to a consumer generator can and should be handled by convention, using existing facilities. -- Greg
On 2010-10-29 09:18, Greg Ewing wrote:
I've been pondering the whole close()-returning-a-value thing I've convinced myself once again that it's a bad idea.
And I still believe we could have made it work. However, I have been doing my own thinking about the whole of PEP 380, PEP 3152, for-loop co-iteration and so on. And I think I have an idea that improves the whole story. The main thing to note is that the expression form of yield-from is mostly intended to make it easier to have cofunctions that return a value, and that there is a problem with reusing StopIteration for that purpose. Now that we have an actual PEP 3152, we could choose to move the necessary support over there. Here is one way to do that: 1) Drop the current PEP 380 support for using "return <value>" inside a generator. That means no extended StopIteration and no expression form of "yield from". And since there are no return values, there is no problem with how "close" should treat them. 2) In PEP 3152, define "return <value>" in a cofunction to raise a new IterationResult exception with the value. (And treat falling off the edge of the function or returning without a value as "return None") 3) In PEP 3152, change the "cocall" expansion so that: <val> = cocall f(*args, **kwargs) Expands to: try: yield from f.__cocall__(*args, **kwargs) except IterationResult as e: <val> = e.value else: raise StopIteration (The same expansion would be used if cocalls are implicit of course). This ensures that a cofunction can raise StopIteration just as a regular function, which means we can extend the iterator protocol to support cofunctions with only minor changes. An interesting variation might be to keep the expression form of yield-from, but change its semantics so that it returns the StopIteration instance that was caught, instead of trying to extract a value. Then adding an IterationResult inheriting from StopIteration and using it for "return <value>" in a generator. That would make all current yield-from examples work with the minor change that the old: <var> = yield from <expr> would need to be written as <var> = (yield from <expr>).value And would have the benefit that the PEP 3152 expansion could reraise the actual StopIteration as in: e = yield from f.__cocall__(*args, **kwargs) if isinstance(e, IterationResult): <var> = e.value else: raise e The idea of returning the exception takes some getting used to, but it solves the problem with StopIteration and cofunctions, and I'm sure I can find some interesting uses for it by itself. Anyway.... Thoughts? - Jacob
On Fri, Oct 29, 2010 at 12:18 AM, Greg Ewing
I've been pondering the whole close()-returning-a-value thing I've convinced myself once again that it's a bad idea.
Essentially the problem is that we're trying to make the close() method, and consequently GeneratorExit, serve two different and incompatible roles.
One role (the one it currently serves) is as an emergency bail-out mechanism. In that role, when we have a stack of generators delegating via yield-from, we want things to behave as thought the GeneratorExit originates in the innermost one and propagates back out of the entire stack. We don't want any of the intermediate generators to catch it and turn it into a StopIteration, because that would give the next outer one the misleading impression that it's business as usual, but it's not.
This seems to be the crux of your objection. But if I look carefully at the expansion in the current version of PEP 380, I don't think this problem actually happens: If the outer generator catches GeneratorExit, it closes the inner generator (by calling its close method, if it exists) and then re-raises the GeneratorExit: except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e I would leave this expansion alone even if g.close() was changed to return the generator's return value. Could it be that you are thinking of your accelerated implementation, which IIRC has a shortcut whereby generator operations (next, send, throw) on the outer generator are *directly* passed to the inner generator when a yield-from is active? It looks to me as if using g.close() to capture the return value of a generator is not of much value when using yield-from, but it can be of value for the simpler pattern that started this thread. Here's an updated version: def gclose(gen): ## Not needed with PEP 380 try: gen.throw(GeneratorExit) except StopIteration as err: return err.args[0] except GeneratorExit: pass # Note: other exceptions are passed out untouched. return None def summer(): total = 0 try: while True: total += yield except GeneratorExit: raise StopIteration(total) ## return total def maxer(): highest = 0 try: while True: value = yield highest = max(highest, value) except GeneratorExit: raise StopIteration(highest) ## return highest def map_to_multiple(it, funcs): gens = [func() for func in funcs] # Create generators for gen in gens: next(gen) # Prime generators for value in it: for gen in gens: gen.send(value) return [gclose(gen) for gen in gens] ## [gen.close() for gen in gens] def main(): print(map_to_multiple(range(100), [summer, maxer])) main() -- --Guido van Rossum (python.org/~guido)
Jacob Holm wrote:
The main thing to note is that the expression form of yield-from is mostly intended to make it easier to have cofunctions that return a value, and that there is a problem with reusing StopIteration for that purpose.
No, I don't think that's where the problem is, if I understand correctly which problem you're referring to. The fact that cofunctions as currently defined in PEP 3152 can't raise StopIteration and have it propagate to the caller is a problem with the use of StopIteration to signal the end of a cofunction. Whether the StopIteration carries a value or not is irrelevant.
And since there are no return values, there is no problem with how "close" should treat them.
There's no problem with that *now*, because close() is currently not defined as returning a value. A problem only arises if we try to overload close() to mean "no more data to send in, give me your result" as well as "bail out now and clean up". And as I pointed out, there are other ways of signalling end of data that work fine with things as they are.
2) In PEP 3152, define "return <value>" in a cofunction to raise a new IterationResult exception with the value.
That would have to apply to *all* forms of return, not just ones with a value.
<var> = (yield from <expr>).value
have the benefit that the PEP 3152 expansion could reraise the actual StopIteration as in:
e = yield from f.__cocall__(*args, **kwargs) if isinstance(e, IterationResult): <var> = e.value else: raise e
There's another way to approach this: define cofunctions so that 'return' in one of its forms is the only way to raise an actual StopIteration, and any explicitly raised StopIteration gets wrapped in something else, such as CoStopIteration. The expansion would then be try: result = yield from f.__cocall__(*args, **kwargs) except CoStopIteration as e: raise e.value where e.value is the original StopIteration instance. This would have the advantage of not requiring any change to yield-from as it stands. -- Greg
On 2010-10-30 01:03, Greg Ewing wrote:
Jacob Holm wrote:
The main thing to note is that the expression form of yield-from is mostly intended to make it easier to have cofunctions that return a value, and that there is a problem with reusing StopIteration for that purpose.
No, I don't think that's where the problem is, if I understand correctly which problem you're referring to. The fact that cofunctions as currently defined in PEP 3152 can't raise StopIteration and have it propagate to the caller is a problem with the use of StopIteration to signal the end of a cofunction.
Exactly.
Whether the StopIteration carries a value or not is irrelevant.
It is relevant if we later want to distinguish between "return" and "raise StopIteration".
And since there are no return values, there is no problem with how "close" should treat them.
There's no problem with that *now*, because close() is currently not defined as returning a value. A problem only arises if we try to overload close() to mean "no more data to send in, give me your result" as well as "bail out now and clean up". And as I pointed out, there are other ways of signalling end of data that work fine with things as they are.
That is what I meant. We were discussing whwther to add a new feature to PEP 380 inspired by having "return <value>" in generators. If we dropped "return <value>" from PEP 380 (with the intent of adding it to PEP 3152 instead), so would the basis for the new feature. End of discussion... AFAICT, adding these features in a consistent way is a lot easier in the context of PEP 3152.
2) In PEP 3152, define "return <value>" in a cofunction to raise a new IterationResult exception with the value.
That would have to apply to *all* forms of return, not just ones with a value.
Of course.
<var> = (yield from <expr>).value
have the benefit that the PEP 3152 expansion could reraise the actual StopIteration as in:
e = yield from f.__cocall__(*args, **kwargs) if isinstance(e, IterationResult): <var> = e.value else: raise e
There's another way to approach this: define cofunctions so that 'return' in one of its forms is the only way to raise an actual StopIteration, and any explicitly raised StopIteration gets wrapped in something else, such as CoStopIteration. The expansion would then be
try: result = yield from f.__cocall__(*args, **kwargs) except CoStopIteration as e: raise e.value
where e.value is the original StopIteration instance.
This would have the advantage of not requiring any change to yield-from as it stands.
That's just ugly... I realize it could work, but I think that makes *both* PEPs more complex than necessary. My suggestion is to cut/change some features from PEP 380 that are in the way and then add them in a cleaner way to PEP 3152. This should simplify both PEPs, at the cost of reopening some of the earlier discussions. - Jacob
Guido van Rossum wrote:
This seems to be the crux of your objection. But if I look carefully at the expansion in the current version of PEP 380, I don't think this problem actually happens: If the outer generator catches GeneratorExit, it closes the inner generator (by calling its close method, if it exists) and then re-raises the GeneratorExit:
Yes, but if you want close() to cause the generator to finish normally, you *don't* want that to happen. You would have to surround the yield-from call with a try block to catch the GeneratorExit, and even then you would lose the return value from the inner generator, which you're probably going to want.
Could it be that you are thinking of your accelerated implementation,
No, not really. The same issues arise either way.
It looks to me as if using g.close() to capture the return value of a generator is not of much value when using yield-from, but it can be of value for the simpler pattern that started this thread.
My concern is that this feature would encourage designing generators with APIs that make it difficult to refactor the implementation using yield-from later on. Simple things don't always stay simple.
def summer(): total = 0 try: while True: total += yield except GeneratorExit: raise StopIteration(total) ## return total
I don't see how this gains you much. The generator is about as complicated either way. The only thing that's simpler is the final step of getting the result, which in my version can be taken care of with a fairly generic helper function that could be provided by the stdlib. -- Greg
Jacob Holm wrote:
It is relevant if we later want to distinguish between "return" and "raise StopIteration".
We want to distinguish between return *without* a value and StopIteration too.
My suggestion is to cut/change some features from PEP 380 that are in the way
But having StopIteration carry a value is *not* one of the things that's in the way, as far as I can see. -- Greg
On Fri, Oct 29, 2010 at 8:09 PM, Greg Ewing
Guido van Rossum wrote:
This seems to be the crux of your objection. But if I look carefully at the expansion in the current version of PEP 380, I don't think this problem actually happens: If the outer generator catches GeneratorExit, it closes the inner generator (by calling its close method, if it exists) and then re-raises the GeneratorExit:
Yes, but if you want close() to cause the generator to finish normally, you *don't* want that to happen. You would have to surround the yield-from call with a try block to catch the GeneratorExit,
Yeah, putting such a try-block around yield from works just as it works around plain yield: it captures the GeneratorExit thrown in. As a bonus, the inner generator is first closed, but the yield-from expression which was interrupted is not completed; just like anything else that raises an exception, execution of the code stops immediately and resumes at the except block.
and even then you would lose the return value from the inner generator, which you're probably going to want.
Really? Can you show a realistic use case? (There was Nick's average-of-sums example but I think nobody liked it.)
Could it be that you are thinking of your accelerated implementation,
No, not really. The same issues arise either way.
Ok.
It looks to me as if using g.close() to capture the return value of a generator is not of much value when using yield-from, but it can be of value for the simpler pattern that started this thread.
My concern is that this feature would encourage designing generators with APIs that make it difficult to refactor the implementation using yield-from later on. Simple things don't always stay simple.
Yeah, but there is also YAGNI. We shouldn't plan every simple thing to become complex; in fact we should expect most simple things to stay simple. Otherwise you'd never use lists and dicts but start with classes right away.
def summer(): total = 0 try: while True: total += yield except GeneratorExit: raise StopIteration(total) ## return total
I don't see how this gains you much. The generator is about as complicated either way.
I'm just concerned about the following:
The only thing that's simpler is the final step of getting the result, which in my version can be taken care of with a fairly generic helper function that could be provided by the stdlib.
In my case too -- it would just be a method on the generator named close(). :-) In addition I like merging use cases that have some overlap, if the non-overlapping parts do not conflict. E.g. I believe the reason we all ended agreeing (at least last year :-) that returning a value should be done through StopIteration was that this makes it so that "return", "return None", "return <value>" and falling of the end of the block are treated uniformly so that equivalences apply both ways. In the case of close(), I *like* that the response to close() can be either cleaning up or returning a value and that close() doesn't care which of the two you do (and in fact it can't tell the difference). -- --Guido van Rossum (python.org/~guido)
On Sat, Oct 30, 2010 at 1:26 PM, Guido van Rossum
Really? Can you show a realistic use case? (There was Nick's average-of-sums example but I think nobody liked it.)
Yeah, I'm much happier with the tally example. It got rid of all the irrelevant framework-y parts :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Guido van Rossum wrote:
On Fri, Oct 29, 2010 at 8:09 PM, Greg Ewing
wrote: and even then you would lose the return value from the inner generator, which you're probably going to want.
Really? Can you show a realistic use case?
Here's an attempt: def variancer(): # Compute variance of values sent in (details left # as an exercise) def stddever(): # Compute standard deviation of values sent in v = yield from variancer() return sqrt(v) -- Greg
On 10/29/2010 10:09 PM, Greg Ewing wrote:
Guido van Rossum wrote:
This seems to be the crux of your objection. But if I look carefully at the expansion in the current version of PEP 380, I don't think this problem actually happens: If the outer generator catches GeneratorExit, it closes the inner generator (by calling its close method, if it exists) and then re-raises the GeneratorExit:
Yes, but if you want close() to cause the generator to finish normally, you *don't* want that to happen. You would have to surround the yield-from call with a try block to catch the GeneratorExit, and even then you would lose the return value from the inner generator, which you're probably going to want.
Ok, after thinking about this for a while, I think the "yield from" would
be too limited if it could only be used for consumers that must run until
the end. That rules out a whole lot of pipes, filters and other things that
consume-some, emit-some, consume-some_more, and emit-some_more.
I think I figured out something that may be more flexible and insn't too
complicated.
The trick is how to tell the "yield from" to stop delegating on a
particular exception. (And be explicit about it!)
# Inside a generator or sub-generator.
...
next(
On Sat, Oct 30, 2010 at 4:42 PM, Ron Adam
Ok, after thinking about this for a while, I think the "yield from" would be too limited if it could only be used for consumers that must run until the end. That rules out a whole lot of pipes, filters and other things that consume-some, emit-some, consume-some_more, and emit-some_more.
Indeed, the "stop-in-the-middle" aspect is tricky, but is the crux of what we're struggling with here.
I think I figured out something that may be more flexible and insn't too complicated.
Basically a way to use yield from, while declaring how to force the end of iteration? Interesting idea. However, I think sentinel values are likely a better way to handle this in a pure PEP 380 context.
Here's an example.
Modifying this example to use sentinel values rather than throwing in exceptions actually makes it all fairly straightforward in a PEP 380 context. So maybe the moral of this whole thread is really "sentinel values good, sentinel exceptions bad". # Helper function to finish off a generator by sending a sentinel value def finish(g, sentinel=None): try: g.send(sentinel) except StopIteration as ex: return ex.value def gtally(end_tally=None): # Tallies numbers until sentinel is passed in count = tally = 0 value = object() while 1: value = yield if value is end_tally: return count, tally count += 1 tally += value def gaverage(end_avg=None): count, tally = (yield from gtally(end_avg)) return tally / count def main(): g = gaverage() next(g) for x in range(100): g.send(x) return finish(g) Even more complex cases, like my sum-of-averages example (or any equivalent multi-level construct) can be implemented without too much hassle, so long as "finish current action" and "start next action" are implemented as two separate steps so the outer layer has a chance to communicate with the outside world before diving into the inner layer. I think we've thrashed this out enough that I, for one, want to see how PEP 380 peforms in the wild as it currently stands before we start tinkering any further. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Oct 29, 2010 at 11:16 PM, Greg Ewing
Guido van Rossum wrote:
On Fri, Oct 29, 2010 at 8:09 PM, Greg Ewing
wrote: and even then you would lose the return value from the inner generator, which you're probably going to want.
Really? Can you show a realistic use case?
Here's an attempt:
def variancer(): # Compute variance of values sent in (details left # as an exercise)
def stddever(): # Compute standard deviation of values sent in v = yield from variancer() return sqrt(v)
Good. I have to get a crazy idea off my chest: maybe the collective hang-up is that GeneratorExit must be special-cased. Let me explore a bit. Take a binary tree node: class Node: def __init__(self, label, left=None, right=None): self.label, self.left, self.right = label, left, right And an inorder traversal function: def inorder(node): if node: yield from inorder(node.left) yield node yield from inorder(node.right) This is a nice example, and different from gtally(), and variance()/stddev(), because of the recursion. Now let's say we want to design a protocol whereby the consumer of the nodes yielded by inorder() can ask the traversal to be stopped. With the code above this is trivial, just call g.close() or throw any other exception in. But now let's first modify inorder() to also return a value computed from the nodes traversed so far. For simplicity I'll use the count: def inorder(node): if not node: return 0 count += yield from inorder(node.left) yield node count += 1 count += yield from inorder(node.right) return count How would we stop this enumeration *and* receive a count of the nodes already enumerated up to that point? Throwing in some exception is the easiest approach. Let's say we throw EOFError. My first attempt has a bug: def inorder(node): if not node: return 0 count = 0 count += yield from inorder(node.left) # Bug here try: count += 1 yield node except EOFError: return count count += yield from inorder(node.right) return count This has the fatal flaw of not responding promptly when the EOFError is caught by the left subtree, since it returns normally and the parent doesn't "see" the EOFError: on the way in it's thrown directly into the first yield-from, on the way out there's no way to distinguish between a regular return or an interrupted one. A potential fix is to return two values: an interrupted flag and a count. But this is pretty ugly (I'm not even going to show the code). A different approach to fixing this is for the throwing code to keep throwing EOFError until the generator stops yielding values: def stop(g): while True: try: g.throw(EOFError) except StopIteration as err: return err.value I'm not concerned with the situation where the generator is already stopped; the EOFError will be bounced out, but that is the caller's problem, as they shouldn't have attempted to stop an already-stopped iterator. (Jacob is probably shaking his head here. :-) This solution doesn't quite work though, because the count returned will include the nodes that were yielded while the stack of generators was winding down. My pragmatic solution for this is to change the protocol so that stopping the generator means that the node yielded last should not be included in the count. If you envision the caller to be running a for-loop, think of calling stop() at the top of the loop rather than at the bottom. (Jacob is now again wondering how they'd get the count if the iterator runs till completion. :-) We can do this by modifying inorder() to bump the count after yielding rather than before: try: yield node except EOFError: return count count += 1 Now, to get back the semantics of getting the correct count *including* the last node seen by the caller, we can modify stop() to advance the generator by one more step: def stop(g): try: next(g) while True: g.throw(EOFError) except StopIteration as err: return err.value This works even if g was positioned after the last item to be yielded: in that case next(g) raises StopIteration. It still doesn't work if we use a for-loop to iterate through the end (Jacob nods :-) but I say they shouldn't be doing that, or they can write a little wrapper for iter() that *does* save the return value from StopIteration. (Okay, half of me says it would be fine to store it on the generator object. :-) [Dramatic pause] [Drumroll] What has this got to do with GeneratorExit and g.close()? I propose to modify g.close() to keep throwing GeneratorExit until the generator stops yielding values, and then capture the return value from StopIteration if that is what was raised. The beauty is then that the PEP 380 expansion can stop special-casing GeneratorExit: it just treats it as every other exception. And stddev() above works! (If you worry about infinite loops: you can get those anyway, by putting "while: True" in an "except GeneratorExit" block. I don't see much reason to worry more in this case.) -- --Guido van Rossum (python.org/~guido)
On 10/30/2010 02:58 AM, Nick Coghlan wrote:
On Sat, Oct 30, 2010 at 4:42 PM, Ron Adam
wrote: Ok, after thinking about this for a while, I think the "yield from" would be too limited if it could only be used for consumers that must run until the end. That rules out a whole lot of pipes, filters and other things that consume-some, emit-some, consume-some_more, and emit-some_more.
Indeed, the "stop-in-the-middle" aspect is tricky, but is the crux of what we're struggling with here.
I think I figured out something that may be more flexible and insn't too complicated.
Basically a way to use yield from, while declaring how to force the end of iteration? Interesting idea.
Not iteration, iteration can continue. It signals the end of delegation, and returns control to the generator that initiated the delegation.
However, I think sentinel values are likely a better way to handle this in a pure PEP 380 context.
Sentinel values aren't always better because they require a extra comparison on each item.
Here's an example.
Modifying this example to use sentinel values rather than throwing in exceptions actually makes it all fairly straightforward in a PEP 380 context. So maybe the moral of this whole thread is really "sentinel values good, sentinel exceptions bad".
# Helper function to finish off a generator by sending a sentinel value def finish(g, sentinel=None): try: g.send(sentinel) except StopIteration as ex: return ex.value
def gtally(end_tally=None): # Tallies numbers until sentinel is passed in count = tally = 0
value = object()
Left over from earlier edit?
while 1: value = yield if value is end_tally: return count, tally count += 1 tally += value
The comparison is executed on every loop. A try-except would be outside the loop.
def gaverage(end_avg=None): count, tally = (yield from gtally(end_avg)) return tally / count
def main(): g = gaverage() next(g) for x in range(100): g.send(x) return finish(g)
Cheers, Ron
Guido van Rossum wrote:
A different approach to fixing this is for the throwing code to keep throwing EOFError until the generator stops yielding values:
That's precisely what I would recommend.
This solution doesn't quite work though, because the count returned will include the nodes that were yielded while the stack of generators was winding down.
My pragmatic solution for this is to change the protocol so that stopping the generator means that the node yielded last should not be included in the count.
This whole example seems contrived to me, so it's hard to say whether this is a good or bad solution.
I propose to modify g.close() to keep throwing GeneratorExit until the generator stops yielding values, and then capture the return value from StopIteration if that is what was raised. The beauty is then that the PEP 380 expansion can stop special-casing GeneratorExit: it just treats it as every other exception.
This was actually suggested during the initial round of discussion, and shot down -- if I remember correctly, on the grounds that it could result in infinite loops. But if you're no longer concerned about that, it's worth considering. My concern is that this would be a fairly substantial change to the intended semantics of close() -- it would no longer be a way of aborting a generator and forcing it to clean up as quickly as possible. But maybe you don't mind losing that functionality? -- Greg
On Sun, Oct 31, 2010 at 1:00 AM, Guido van Rossum
What has this got to do with GeneratorExit and g.close()? I propose to modify g.close() to keep throwing GeneratorExit until the generator stops yielding values, and then capture the return value from StopIteration if that is what was raised. The beauty is then that the PEP 380 expansion can stop special-casing GeneratorExit: it just treats it as every other exception. And stddev() above works! (If you worry about infinite loops: you can get those anyway, by putting "while: True" in an "except GeneratorExit" block. I don't see much reason to worry more in this case.)
I'm back to liking your general idea, but wanting to use a new method and exception for the task to keep the two sets of semantics orthogonal :) If we add a finish() method that corresponds to your stop() function, and a GeneratorReturn exception as a peer to GeneratorExit: class GeneratorReturn(BaseException): pass def finish(self): if g.gi_frame is None: return self._result # (or raise RuntimeError) try: next(self) while True: self.throw(GeneratorReturn) except StopIteration as ex: return ex.value Then your node counter iterator (nice example, btw) would simply look like: def count_nodes(node): if not node: return 0 count = 0 count += yield from count_nodes(node.left) try: yield node except GeneratorReturn: return count count += 1 # Only count nodes when next is called in response count += yield from count_nodes(node.right) return count Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Oct 31, 2010 at 2:54 AM, Ron Adam
However, I think sentinel values are likely a better way to handle this in a pure PEP 380 context.
Sentinel values aren't always better because they require a extra comparison on each item.
Yep, Guido's example made me realise I was wrong on that front. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10/30/2010 07:35 PM, Nick Coghlan wrote:
On Sun, Oct 31, 2010 at 2:54 AM, Ron Adam
wrote: However, I think sentinel values are likely a better way to handle this in a pure PEP 380 context.
Sentinel values aren't always better because they require a extra comparison on each item.
Yep, Guido's example made me realise I was wrong on that front.
BTW: A sentinal could still work, and the 'except <exception>' could be optional. The finish function isn't needed in this one. def gtally(end_tally): # Tallies numbers until sentinel is passed in count = tally = 0 while 1: value = yield if value is end_tally: break count += 1 tally += value yield count, tally def gaverage(end_avg): yield from gtally(end_avg) yield tally / count def main(): g = gaverage(None) next(g) for x in range(100): g.send(x) return g.send(None) Using sentinels not always wrong either. The data may have natural sentinel values in it. In those cases, value testing is what you want. I would like to be able to do it both ways myself. :-) Cheers, Ron
On Sat, Oct 30, 2010 at 5:09 PM, Greg Ewing
Guido van Rossum wrote:
A different approach to fixing this is for the throwing code to keep throwing EOFError until the generator stops yielding values:
That's precisely what I would recommend.
This solution doesn't quite work though, because the count returned will include the nodes that were yielded while the stack of generators was winding down.
My pragmatic solution for this is to change the protocol so that stopping the generator means that the node yielded last should not be included in the count.
This whole example seems contrived to me, so it's hard to say whether this is a good or bad solution.
I tried to come up with an example that made non-trivial use of yield-from. Iterating over a binary tree is about the most natural example I could think of. The desire to interrupt the iteration in the middle also seems natural. I agree that the addition of a return value for the whole generator is somewhat contrived. Maybe a better value to return would have been the maximum depth encountered?
I propose to modify g.close() to keep throwing GeneratorExit until the generator stops yielding values, and then capture the return value from StopIteration if that is what was raised. The beauty is then that the PEP 380 expansion can stop special-casing GeneratorExit: it just treats it as every other exception.
This was actually suggested during the initial round of discussion, and shot down -- if I remember correctly, on the grounds that it could result in infinite loops. But if you're no longer concerned about that, it's worth considering.
My concern is that this would be a fairly substantial change to the intended semantics of close() -- it would no longer be a way of aborting a generator and forcing it to clean up as quickly as possible.
But maybe you don't mind losing that functionality?
I've thought about this long and hard, and in the end I do mind losing the special semantics of GeneratorExit. I am withdrawing my proposal. I don't think the PEP should be weighed down with a separate method + exception to request a return value either -- rather, a framework that needs such functionality can create a local convention. I doubt that the intermingling of frameworks will be such that that is much of a burden for framework users -- if it is, we can always write a PEP to standardize such a convention later. With this, I think PEP 380 can be accepted pretty much as is, although it would be nice if a part of my example made it into the PEP -- I don't think there is anything wrong with PEPs having examples. -- --Guido van Rossum (python.org/~guido)
participants (5)
-
Greg Ewing
-
Guido van Rossum
-
Jacob Holm
-
Nick Coghlan
-
Ron Adam