PEP 342 suggestion: start(), __call__() and unwind_call() methods
I'm lifting Jason's PEP 342 suggestions out of the recent PEP 343 thread, in case some of the folks interested in coroutines stopped following that discussion. Jason suggested two convenience methods, .start() and .finish(). start() simply asserted that the generator hadn't been started yet, and I find the parallel with "Thread.start()" appealing: def start(self): """ Convenience method -- exactly like next(), but assert that this coroutine hasn't already been started. """ if self.__started: raise RuntimeError("Coroutine already started") return self.next() I've embellished Jason's suggested finish() method quite a bit though. 1. Use send() rather than next() 2. Call it __call__() rather than finish() 3. Add an unwind_call() variant that gives similar semantics for throw() 4. Support getting a return value from the coroutine using the syntax "raise StopIteration(val)" 5. Add an exception "ContinueIteration" that is used to indicate the generator hasn't finished yet, rather than expecting the generator to finish and raising RuntimeError if it doesn't It ends up looking like this: def __call__(self, value=None): """ Call a generator as a coroutine Returns the first argument supplied to StopIteration or None if no argument was supplied. Raises ContinueIteration with the value yielded as the argument if the generator yields a value """ if not self.__started: raise RuntimeError("Coroutine not started") try: if exc: yield_val = self.throw(value, *exc) else: yield_val = self.send(value) except (StopIteration), ex: if ex.args: return args[0] else: raise ContinueIteration(yield_val) def unwind_call(self, *exc): """Raise an exception in a generator used as a coroutine. Returns the first argument supplied to StopIteration or None if no argument was supplied. Raises ContinueIteration if the generator yields a value with the value yield as the argument """ try: yield_val = self.throw(*exc) except (StopIteration), ex: if ex.args: return args[0] else: raise ContinueIteration(yield_val) Now here's the trampoline scheduler from PEP 342 using this idea: import collections class Trampoline: """Manage communications between coroutines""" running = False def __init__(self): self.queue = collections.deque() def add(self, coroutine): """Request that a coroutine be executed""" self.schedule(coroutine) def run(self): result = None self.running = True try: while self.running and self.queue: func = self.queue.popleft() result = func() return result finally: self.running = False def stop(self): self.running = False def schedule(self, coroutine, stack=(), call_result=None, *exc): # Define the new pseudothread def pseudothread(): try: if exc: result = coroutine.unwind_call(call_result, *exc) else: result = coroutine(call_result) except (ContinueIteration), ex: # Called another coroutine callee = ex.args[0] self.schedule(callee, (coroutine,stack)) except: if stack: # send the error back to the caller caller = stack[0] prev_stack = stack[1] self.schedule( caller, prev_stack, *sys.exc_info() ) else: # Nothing left in this pseudothread to # handle it, let it propagate to the # run loop raise else: if stack: # Finished, so pop the stack and send the # result to the caller caller = stack[0] prev_stack = stack[1] self.schedule(caller, prev_stack, result) # Add the new pseudothread to the execution queue self.queue.append(pseudothread) Notice how a non-coroutine callable can be yielded, and it will still work happily with the scheduler, because the desire to continue execution is indicated by the ContinueIteration exception, rather than by the type of the returned value. With this relatively dumb scheduler, that doesn't provide any particular benefit - the specific pseudothread doesn't block, but eventually the scheduler itself blocks when it executes the non-coroutine call. However, it wouldn't take too much to make the scheduler smarter and give it a physical thread pool that it used whenever it encountered a non-coroutine call And that's the real trick here: with these additions to PEP 342, the decision of how to deal with blocking calls could be made in the scheduler, without affecting the individual coroutines. All the coroutine writers would need to remember to do is to write any potentially blocking operations as yielded lambda expressions or functional.partial invocations, rather than as direct function calls. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Nick Coghlan wrote:
It ends up looking like this:
def __call__(self, value=None): """ Call a generator as a coroutine
Returns the first argument supplied to StopIteration or None if no argument was supplied. Raises ContinueIteration with the value yielded as the argument if the generator yields a value """ if not self.__started: raise RuntimeError("Coroutine not started") try: if exc: yield_val = self.throw(value, *exc) else: yield_val = self.send(value) except (StopIteration), ex: if ex.args: return args[0] else: raise ContinueIteration(yield_val)
Oops, I didn't finish fixing this after I added unwind_call(). Try this version instead: def __call__(self, value=None): """ Call a generator as a coroutine Returns the first argument supplied to StopIteration or None if no argument was supplied. Raises ContinueIteration with the value yielded as the argument if the generator yields a value """ try: yield_val = self.send(value) except (StopIteration), ex: if ex.args: return args[0] else: raise ContinueIteration(yield_val) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
At 09:50 PM 10/7/2005 +1000, Nick Coghlan wrote:
Notice how a non-coroutine callable can be yielded, and it will still work happily with the scheduler, because the desire to continue execution is indicated by the ContinueIteration exception, rather than by the type of the returned value.
Whaaaa? You raise an exception to indicate the *normal* case? That seems, um... well, a Very Bad Idea. I also don't see any point to start(), or understand what finish() does or why you'd want it. Last, but far from least, as far as I can tell you can implement all of these semantics using PEP 342 as it sits. That is, it's very simple to make decorators or classes that add those semantics. I don't see anything that requires them to be part of Python.
Phillip J. Eby wrote:
At 09:50 PM 10/7/2005 +1000, Nick Coghlan wrote:
Notice how a non-coroutine callable can be yielded, and it will still work happily with the scheduler, because the desire to continue execution is indicated by the ContinueIteration exception, rather than by the type of the returned value.
Whaaaa? You raise an exception to indicate the *normal* case? That seems, um... well, a Very Bad Idea.
The sheer backwardness of my idea occurred to me after I'd got some sleep :)
Last, but far from least, as far as I can tell you can implement all of these semantics using PEP 342 as it sits. That is, it's very simple to make decorators or classes that add those semantics. I don't see anything that requires them to be part of Python.
Yeah, I've now realised that you can do all of this more simply by doing it directly in the scheduler using StopIteration to indicate when the coroutine is done, and using yield to indicate "I'm not done yet". So with a bit of thought, I came up with a scheduler that has all the benefits I described, and only uses the existing PEP 342 methods. When writing a coroutine for this scheduler, you can do 6 things via the scheduler: 1. Raise StopIteration to indicate "I'm done" and return None to your caller 2. Raise StopIteration with a single argument to return a value other than None to your caller 3. Raise a different exception and have that exception propagate up to your caller 5. Yield None to allow other coroutines to be executed 5. Yield a coroutine to request a call to that coroutine 6. Yield a callable to request an asynchronous call using that object Yielding anything else, or trying to raise StopIteration with more than one argument results in a TypeError being raised *at the point of the offending yield or raise statement*, rather than taking out the scheduler itself. The more I explore the possibilities of PEP 342, the more impressed I am by the work that went into it! Cheers, Nick. P.S. Here's the Trampoline scheduler described above: import collections class Trampoline: """Manage communications between coroutines""" running = False def __init__(self): self.queue = collections.deque() def add(self, coroutine): """Request that a coroutine be executed""" self.schedule(coroutine) def run(self): result = None self.running = True try: while self.running and self.queue: func = self.queue.popleft() result = func() return result finally: self.running = False def stop(self): self.running = False def schedule(self, coroutine, stack=(), call_result=None, *exc): # Define the new pseudothread def pseudothread(): try: if exc: callee = coroutine.throw(call_result, *exc) else: callee = coroutine(call_result) except (StopIteration), ex: # Coroutine finished cleanly if stack: # Send the result to the caller caller = stack[0] prev_stack = stack[1] if len(ex.args) > 1: # Raise a TypeError in the current coroutine self.schedule(coroutine, stack, TypeError, "Too many arguments to StopIteration" ) elif ex.args: self.schedule(caller, prev_stack, ex.args[0]) else: self.schedule(caller, prev_stack) except: # Coroutine finished with an exception if stack: # send the error back to the caller caller = stack[0] prev_stack = stack[1] self.schedule( caller, prev_stack, *sys.exc_info() ) else: # Nothing left in this pseudothread to # handle it, let it propagate to the # run loop raise else: # Coroutine isn't finished yet if callee is None: # Reschedule the current coroutine self.schedule(coroutine, stack) elif isinstance(callee, types.GeneratorType): # Requested a call to another coroutine self.schedule(callee, (coroutine,stack)) elif callable(callee): # Requested an asynchronous call self._make_async_call(callee, coroutine, stack) else: # Raise a TypeError in the current coroutine self.schedule(coroutine, stack, TypeError, "Illegal argument to yield" ) # Add the new pseudothread to the execution queue self.queue.append(pseudothread) def _make_async_call(self, blocking_call, caller, stack): # Assume @threaded decorator takes care of # - returning a function with a call method which # kick starts the function execution and returns # a Future object to give access to the result. # - farming call out to a physical thread pool # - keeping the Thread object executing the async # call alive after this function exits @threaded def async_call(): try: result = blocking_call() except: # send the error back to the caller self.schedule( caller, stack, *sys.exc_info() ) else: # Send the result back to the caller self.schedule(caller, stack, result) # Start the asynchronous call async_call() -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Nick Coghlan wrote:
Phillip J. Eby wrote:
Nick Coghlan wrote:
[...]
Last, but far from least, as far as I can tell you can implement all of these semantics using PEP 342 as it sits. That is, it's very simple to make decorators or classes that add those semantics. I don't see anything that requires them to be part of Python.
Yeah, I've now realised that you can do all of this more simply by doing it directly in the scheduler using StopIteration to indicate when the coroutine is done, and using yield to indicate "I'm not done yet".
Earlier this week, i proposed legalizing "return Result" inside a generator, and making it act like "raise StopIteration( Result )", for exactly this reason. IMHO, this is an elegant and straightforward extension of the current semantics of returns inside generators, and is the final step toward making generator-based concurrent tasks[1] look just like the equivalent synchronous code (with the only difference, more-or-less, being the need for appropriate "yield" keywords, and a task runner/scheduler loop). This change would make a huge difference to the practical usability of these generator-based tasks. I think they're much less likely to catch on if you have to write "raise StopIteration( Result )" (or "_return( Result )") all the time. [1] a.k.a. coroutines, which i don't think is an accurate name, anymore.
On 10/7/05, Piet Delport <pjd@satori.za.net> wrote:
Earlier this week, i proposed legalizing "return Result" inside a generator, and making it act like "raise StopIteration( Result )", for exactly this reason.
IMHO, this is an elegant and straightforward extension of the current semantics of returns inside generators, and is the final step toward making generator-based concurrent tasks[1] look just like the equivalent synchronous code (with the only difference, more-or-less, being the need for appropriate "yield" keywords, and a task runner/scheduler loop).
This change would make a huge difference to the practical usability of these generator-based tasks. I think they're much less likely to catch on if you have to write "raise StopIteration( Result )" (or "_return( Result )") all the time.
[1] a.k.a. coroutines, which i don't think is an accurate name, anymore.
Before we do this I'd like to see you show some programming examples that show how this would be used. I'm having a hard time understanding where you would need this but I realize I haven't used this paradigm enough to have a good feel for it, so I'm open for examples. At least this makes more sense than mapping "return X" into "yield X; return" as someone previously proposed. :) -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
This change would make a huge difference to the practical usability of these generator-based tasks. I think they're much less likely to catch on if you have to write "raise StopIteration( Result )" (or "_return( Result )") all the time.
[1] a.k.a. coroutines, which i don't think is an accurate name, anymore.
Before we do this I'd like to see you show some programming examples that show how this would be used. I'm having a hard time understanding where you would need this but I realize I haven't used this paradigm enough to have a good feel for it, so I'm open for examples.
At least this makes more sense than mapping "return X" into "yield X; return" as someone previously proposed. :)
It would be handy when the generators are being used as true pseudothreads with a scheduler like the one I posted earlier in this discussion. It allows these pseudothreads to "call" each other by yielding the call as a lambda or partial function application that produces a zero-argument callable. The called pseudothread can then yield as many times as it wants (either making its own calls, or just being a well-behaved member of a cooperatively MT environment), and then finally returning the value that the original caller requested. Using 'return' for this is actually a nice idea, and if we ever do make it legal to use 'return' in generators, these are the semantics it should have. However, I'm not sure its something we should be adding *right now* as part of PEP 342 - writing "raise StopIteration" and "raise StopIteration(result)", and saying that a generator includes an implied "raise StopIteration" after its last line of code really isn't that difficult to understand, and is completely explicit about what is going on. My basic concern is that I think replacing "raise StopIteration" with "return" and "raise StopIteration(EXPR)" with "return EXPR" would actually make such code easier to write at the expense of making it harder to *read*, because the fact that an exception is being raised is obscured. Consider the following two code snippets: def function(): try: return except StopIteration: print "We never get here." def generator(): yield try: return except StopIteration: print "But we would get here!" So, instead of having "return" automatically map to "raise StopIteration" inside generators, I'd like to suggest we keep it illegal to use "return" inside a generator, and instead add a new attribute "result" to StopIteration instances such that the following three conditions hold: # Result is None if there is no argument to StopIteration try: raise StopIteration except StopIteration, ex: assert ex.result is None # Result is the argument if there is exactly one argument try: raise StopIteration(expr) except StopIteration, ex: assert ex.result == ex.args[0] # Result is the argument tuple if there are multiple arguments try: raise StopIteration(expr1, expr2) except StopIteration, ex: assert ex.result == ex.args This precisely parallels the behaviour of return statements: return # Call returns None return expr # Call returns expr return expr1, expr2 # Call returns (expr1, expr2) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Guido van Rossum wrote:
Before we do this I'd like to see you show some programming examples that show how this would be used. I'm having a hard time understanding where you would need this but I realize I haven't used this paradigm enough to have a good feel for it, so I'm open for examples.
On 10/8/05, Nick Coghlan <ncoghlan@gmail.com> wrote:
It would be handy when the generators are being used as true pseudothreads with a scheduler like the one I posted earlier in this discussion. It allows these pseudothreads to "call" each other by yielding the call as a lambda or partial function application that produces a zero-argument callable. The called pseudothread can then yield as many times as it wants (either making its own calls, or just being a well-behaved member of a cooperatively MT environment), and then finally returning the value that the original caller requested.
Using 'return' for this is actually a nice idea, and if we ever do make it legal to use 'return' in generators, these are the semantics it should have.
However, I'm not sure its something we should be adding *right now* as part of PEP 342 - writing "raise StopIteration" and "raise StopIteration(result)", and saying that a generator includes an implied "raise StopIteration" after its last line of code really isn't that difficult to understand, and is completely explicit about what is going on.
My basic concern is that I think replacing "raise StopIteration" with "return" and "raise StopIteration(EXPR)" with "return EXPR" would actually make such code easier to write at the expense of making it harder to *read*, because the fact that an exception is being raised is obscured. Consider the following two code snippets:
def function(): try: return except StopIteration: print "We never get here."
def generator(): yield try: return except StopIteration: print "But we would get here!"
Right. Plus, Piet also remarked that the value is silently ignored when the generator is used in a for-loop. Since that's likely to be the majority of generators, I'd worry that accepting "return X" would increase the occurrence of bugs caused by someone habitually writing "return X" where they meant "yield X". (Assuming there's another yield in the generator, otherwise it wouldn't be a generator and the error would reveal itself very differently.)
So, instead of having "return" automatically map to "raise StopIteration" inside generators, I'd like to suggest we keep it illegal to use "return" inside a generator, and instead add a new attribute "result" to StopIteration instances such that the following three conditions hold:
# Result is None if there is no argument to StopIteration try: raise StopIteration except StopIteration, ex: assert ex.result is None
# Result is the argument if there is exactly one argument try: raise StopIteration(expr) except StopIteration, ex: assert ex.result == ex.args[0]
# Result is the argument tuple if there are multiple arguments try: raise StopIteration(expr1, expr2) except StopIteration, ex: assert ex.result == ex.args
This precisely parallels the behaviour of return statements: return # Call returns None return expr # Call returns expr return expr1, expr2 # Call returns (expr1, expr2)
This seems a bit overdesigned; I'd expect that the trampoline scheduler could easily enough pick the args tuple apart to get the same effect without adding another attribute unique to StopIteration. I'd like to keep StopIteration really lightweight so it doesn't slow down its use in other places. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On 10/8/05, Nick Coghlan <ncoghlan@gmail.com> wrote:
So, instead of having "return" automatically map to "raise StopIteration" inside generators, I'd like to suggest we keep it illegal to use "return" inside a generator, and instead add a new attribute "result" to StopIteration instances such that the following three conditions hold:
# Result is None if there is no argument to StopIteration try: raise StopIteration except StopIteration, ex: assert ex.result is None
# Result is the argument if there is exactly one argument try: raise StopIteration(expr) except StopIteration, ex: assert ex.result == ex.args[0]
# Result is the argument tuple if there are multiple arguments try: raise StopIteration(expr1, expr2) except StopIteration, ex: assert ex.result == ex.args
This precisely parallels the behaviour of return statements: return # Call returns None return expr # Call returns expr return expr1, expr2 # Call returns (expr1, expr2)
This seems a bit overdesigned; I'd expect that the trampoline scheduler could easily enough pick the args tuple apart to get the same effect without adding another attribute unique to StopIteration. I'd like to keep StopIteration really lightweight so it doesn't slow down its use in other places.
True. And it would be easy enough for a framework to have a utility function that looked like: def getresult(ex): args = ex.args if not args: return None elif len(args) == 1: return args[0] else: return args Although, if StopIteration.result was a read-only property with the above definition, wouldn't that give us the benefit of "one obvious way" to return a value from a coroutine without imposing any runtime cost on normal use of StopIteration to finish an iterator? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Nick Coghlan wrote:
Although, if StopIteration.result was a read-only property with the above definition, wouldn't that give us the benefit of "one obvious way" to return a value from a coroutine without imposing any runtime cost on normal use of StopIteration to finish an iterator?
Sometimes I miss the obvious. There's a *much*, *much* better place to store the return value of a generator than on the StopIteration exception that it raises when it finishes. Just save the return value in the *generator*. And then provide a method on generators that is the functional equivalent of: def result(): # Finish the generator if it isn't finished already for step in self: pass return self._result # Return the result saved when the block finished It doesn't matter that a for loop swallows the StopIteration exception any more, because the return value is retrieved directly from the generator. I also like that this interface could still be used even if the work of getting the result is actually farmed off to a separate thread or process behind the scenes. Cheers, Nick. P.S. Here's what a basic trampoline scheduler without builtin asynchronous call support would look like if coroutines could return values directly. The bits that it cleans up are marked "NEW": import collections class Trampoline: """Manage communications between coroutines""" running = False def __init__(self): self.queue = collections.deque() def add(self, coroutine): """Request that a coroutine be executed""" self.schedule(coroutine) def run(self): result = None self.running = True try: while self.running and self.queue: func = self.queue.popleft() result = func() return result finally: self.running = False def stop(self): self.running = False def schedule(self, coroutine, stack=(), call_result=None, *exc): # Define the new pseudothread def pseudothread(): try: if exc: callee = coroutine.throw(call_result, *exc) else: callee = coroutine.send(call_result) except StopIteration: # NEW: no need to name exception # Coroutine finished cleanly if stack: # Send the result to the caller caller = stack[0] prev_stack = stack[1] # NEW: get result directly from callee self.schedule( caller, prev_stack, callee.result() ) except: # Coroutine finished with an exception if stack: # send the error back to the caller caller = stack[0] prev_stack = stack[1] self.schedule( caller, prev_stack, *sys.exc_info() ) else: # Nothing left in this pseudothread to # handle it, let it propagate to the # run loop raise else: # Coroutine isn't finished yet if callee is None: # Reschedule the current coroutine self.schedule(coroutine, stack) elif isinstance(callee, types.GeneratorType): # Make a call to another coroutine self.schedule(callee, (coroutine,stack)) elif iscallable(callee): # Make a blocking call in a separate thread self.schedule( threaded(callee), (coroutine,stack) ) else: # Raise a TypeError in the current coroutine self.schedule(coroutine, stack, TypeError, "Illegal argument to yield" ) # Add the new pseudothread to the execution queue self.queue.append(pseudothread) P.P.S. Here's the simple coroutine that threads out a call to support asynchronous calls with the above scheduler: def threaded(func): class run_func(threading.Thread): def __init__(self): super(run_func, self).__init__() self.finished = False def run(self): print "Making call" self.result = func() self.finished = True print "Made call" call = run_func() call.start() print "Started call" while not call.finished: yield # Not finished yet so reschedule print "Finished call" return call.result I tried this out by replacing 'yield' with 'yield None' and 'return call.result' with 'print call.result': Py> x = threaded(lambda: "Hi there!") Py> x.next() Started call Making call Made call Py> x.next() Finished call Hi there! Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
On 10/9/05, Nick Coghlan <ncoghlan@gmail.com> wrote:
Sometimes I miss the obvious. There's a *much*, *much* better place to store the return value of a generator than on the StopIteration exception that it raises when it finishes. Just save the return value in the *generator*.
And then provide a method on generators that is the functional equivalent of:
def result(): # Finish the generator if it isn't finished already for step in self: pass return self._result # Return the result saved when the block finished
It doesn't matter that a for loop swallows the StopIteration exception any more, because the return value is retrieved directly from the generator.
Actually, I don't like this at all. It harks back to earlier proposals where state was stored on the generator (e.g. PEP 288).
I also like that this interface could still be used even if the work of getting the result is actually farmed off to a separate thread or process behind the scenes.
That seems an odd use case for generators, better addressed by creating an explicit helper object when the need exists. I bet that object will need to exist anyway to hold other information related to the exchange of information between threads (like a lock or a Queue). Looking at your example, I have to say that I find the trampoline example from PEP 342 really hard to understand. It took me several days to get it after Phillip first put it in the PEP, and that was after having reconstructed the same functionality independently. (I have plans to replace or augment it with a different set of examples, but haven't gotten the time. Old story...) I don't think that something like that ought to be motivating generator extensions. I also think that using a thread for async I/O is the wrong approach -- if you wanted to use threads shou should be using threads and you wouldn't be dealing with generators. There's a solution that uses select() which can handle as many sockets as you want without threads and without the clumsy polling ("is it ready yet? is it ready yet? is it ready yet?"). I urge you to leave well enough alone. There's room for extensions after people have built real systems with the raw material provided by PEP 342 and 343. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Le dimanche 09 octobre 2005 à 07:46 -0700, Guido van Rossum a écrit :
I also think that using a thread for async I/O is the wrong approach -- if you wanted to use threads shou should be using threads and you wouldn't be dealing with generators. There's a solution that uses select() which can handle as many sockets as you want without threads and without the clumsy polling
select() works with sockets. But nothing else if you want to stay cross-platform, so async file IO and other things remain open questions. By the way, you don't need clumsy polling to wait for helper threads ;) You can just use a ConditionVariable from the threading package (or something else with the same semantics). BTW, I'm not arguing at all for the extension proposal. Integrating async stuff into generators does not need an API extension IMO. I'm already doing it in my scheduler. An example which just waits for an external command to finish and periodically spins a character in the meantime: http://svn.berlios.de/viewcvs/tasklets/trunk/examples/popen1.py?view=markup The scheduler code is here: http://svn.berlios.de/viewcvs/tasklets/trunk/softlets/core/switcher.py?view=... Regards Antoine.
Nick Coghlan wrote:
Sometimes I miss the obvious. There's a *much*, *much* better place to store the return value of a generator than on the StopIteration exception that it raises when it finishes. Just save the return value in the *generator*.
I'm not convinced that this is better, because it would make value-returning something specific to generators. On the other hand, raising StopIteration(value) is something that any iterator can easily do, whether it's implemented as a generator, a Python class, a C type, or whatever. Besides, it doesn't smell right to me -- sort of like returning a value from a function by storing it in a global rather than using a return statement. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
Greg Ewing wrote:
Nick Coghlan wrote:
Sometimes I miss the obvious. There's a *much*, *much* better place to store the return value of a generator than on the StopIteration exception that it raises when it finishes. Just save the return value in the *generator*.
I'm not convinced that this is better, because it would make value-returning something specific to generators.
On the other hand, raising StopIteration(value) is something that any iterator can easily do, whether it's implemented as a generator, a Python class, a C type, or whatever.
Besides, it doesn't smell right to me -- sort of like returning a value from a function by storing it in a global rather than using a return statement.
Yeah, the various responses have persuaded me that having generators resemble threads in that they don't have a defined "return value" isn't a bad thing at all. Although that means I've gone all the way back to preferring the status quo - if you want to pass data back from a generator when it terminates, just use StopIteration(result). I'm starting to think we want to let PEP 342 bake for at least one release cycle before deciding what (if any) additional behaviour should be added to generators. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
On 10/10/05, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm starting to think we want to let PEP 342 bake for at least one release cycle before deciding what (if any) additional behaviour should be added to generators.
Yes please! -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Plus, Piet also remarked that the value is silently ignored when the generator is used in a for-loop. ... I'd worry that accepting "return X" would increase the occurrence of bugs caused by someone habitually writing "return X" where they meant "yield X".
Then have for-loops raise an exception if they get a StopIteration with something other than None as an argument.
I'd like to keep StopIteration really lightweight so it doesn't slow down its use in other places.
You could leave StopIteration itself alone altogether and have a subclass StopIterationWithValue for returning things. This would make the for-loop situation even safer, since then you could distinguish between falling off the end of a generator and executing 'return None' inside it. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
On Oct 8, 2005, at 9:10 PM, Nick Coghlan wrote:
So, instead of having "return" automatically map to "raise StopIteration" inside generators, I'd like to suggest we keep it illegal to use "return" inside a generator
Only one issue with that: it's _not currently illegal_ to use return inside a generator. From the view of the outsider, it currently effectively does currently map to "raise StopIteration". But not on the inside, just like you'd expect to happen. The only proposed change to the semantics is to also allow a value to be provided with the return.
def generator(): yield try: return except StopIteration: print "But we would get here!"
def generator(): ... yield 5 ... try: ... return ... except StopIteration: ... print "But we would get here!" ... x=generator() x.next() 5 x.next() Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration
James
James Y Knight wrote:
On Oct 8, 2005, at 9:10 PM, Nick Coghlan wrote:
So, instead of having "return" automatically map to "raise StopIteration" inside generators, I'd like to suggest we keep it illegal to use "return" inside a generator
Only one issue with that: it's _not currently illegal_ to use return inside a generator. From the view of the outsider, it currently effectively does currently map to "raise StopIteration". But not on the inside, just like you'd expect to happen. The only proposed change to the semantics is to also allow a value to be provided with the return.
Huh. I'd have sworn I'd tried that and it didn't work. Maybe I was using a value with the return, and had forgotten the details of the error message. In that case, I have far less of an objection to the idea - particularly since it *does* forcibly terminate the generator's block without triggering any exception handlers. I was forgetting that the StopIteration exception is actually raised external to the generator code block - it's created by the surrounding generator object once the code block terminates. That means the actual change being proposed is smaller than I thought: 1. Change the compiler to allow an argument to return inside a generator 2. Change generator objects to use the value returned by their internal code block as the argument to the StopIteration exception they create if the block terminates Note that this would change the behaviour of normal generators - they will raise "StopIteration(None)", rather than the current "StopIteration()". I actually kind of like that - it means that generators become even more like functions, with their return value being held in ex.args[0]. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
participants (8)
-
Antoine Pitrou -
Greg Ewing -
Guido van Rossum -
James Y Knight -
Nick Coghlan -
Nick Coghlan -
Phillip J. Eby -
Piet Delport