Exceptions thrown from generators.. patch.
I was able to create a patch for testing this idea. The hard part was in getting to know cpython well enough to do it. :-) To get it to work, I made the following change in ceval.c so that the main loop will accept a pointer rather than an int for the throwflag. That allows an opcode to set it before it yields an exception. ie... a reverse throw. PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) { return PyEval_EvalFrame_Ex(f, &throwflag); } PyObject * PyEval_EvalFrame_Ex(PyFrameObject *f, int *throwflag) { ... TARGET(YIELD_EXCEPT) *throwflag = 1; retval = POP(); f->f_stacktop = stack_pointer; why = WHY_YIELD; goto fast_yield; ... The genobject gen_send_ex() function checks the throwflag value after a send to see if it got a thrown out exception back. (Rather than one yielded out.) A new keyword 'throws' was needed to go with the YIELD_EXCEPT opcode. I didn't see anyway to do it with a function or method. It should be possible to set the exception in the ceval loop rather than yielding it out, but I think that would be more complex. Because the exception isn't raised inside the generator code object, but is yielded out first, the generator can be continued as if it was a regular yielded value. No magic required, and no fiddling with the exception stack was needed. It doesn't effect unexpected exceptions, or exceptions raised with raise. Those will terminate a generator as always. Python 3.3.0a0 (qbase qtip tip yield_except:92ac2848438f+, Nov 18 2011, 21:59:57) [GCC 4.6.1] on linux Type "help", "copyright", "credits" or "license" for more information.
The three main benefits of being able to do this are... * To use switch like exception structures for flow control in schedulers and coroutines. * For consumer type coroutines to be able reject and re-request data in a nice way without terminating. ie.. the reverse of throwing in an exception in, in order to change what a generator does. * It creates alternative channels for data input and output by being able to both throw exceptions in and out of generators. Those signals can carry objects in and out and not burden the fast yield data path with testing for special wrapper objects. Here's an example of it being used in a simple scheduler. ----------- class Suspend(Exception): pass def Person(name, count, mode): n = 0 while n < count: if mode == 0: # The normal data path. yield name, count else: # Use an exception as an alternative data path. throws Suspend(name, count) n += 1 # return raise StopIteration(name, n) def main(data, mode): stack = [Person(*(args + (mode,))) for args in data] results = [] while stack: done = [] for ct in stack: try: print('yield', *next(ct)) # from yield except Suspend as exc: print('throws', *exc.args) # from throws except StopIteration as exc: results.append(exc.args) continue done.append(ct) stack = done print(results) return results if __name__ == "__main__": data = [("John", 2), ("Micheal", 3), ("Terry", 4)] results1 = main(data, 0) results2 = main(data, 1) assert(results1 == results2 == data) ------------- The output looks like... yield John 2 yield Micheal 3 yield Terry 4 yield John 2 yield Micheal 3 yield Terry 4 yield Micheal 3 yield Terry 4 yield Terry 4 [('John', 2), ('Micheal', 3), ('Terry', 4)] throws John 2 throws Micheal 3 throws Terry 4 throws John 2 throws Micheal 3 throws Terry 4 throws Micheal 3 throws Terry 4 throws Terry 4 [('John', 2), ('Micheal', 3), ('Terry', 4)] This shows that 'throws' works a lot like 'yield'. Open issues: * A better name than 'throws' might be good. * Should it get the object sent in. <object> = throws <exception> Or should it be ... throws <exception> * What would be the best argument form.. Should it take the same arguments as raise or just a single expression. Python's test suite passes as this doesn't change anything that already works. I haven't tested it with the yield-from patch yet, but I think if it can throw out exceptions in the same way yield-from yields out, that it will make some things easier and nicer to do. If anyone is interested, I can create a tracker item and put the patch there where it can be improved further. Cheers, Ron
On Fri, Nov 18, 2011 at 10:24 PM, Ron Adam <ron3200@gmail.com> wrote:
neat!
Open issues:
* A better name than 'throws' might be good.
I don't like adding another keyword or confusing things by adding the "throw" verb to a language that already firmly uses the verb "raise" when speaking of exceptions. A double word syntax might make sense here. It'd be good to keep 'yeild' in it to make it clear that this is offering the exception out in a non-terminal manner. yield raise MyException(x,y,z) or if you've caught an exception from something and want to pass that on to the caller without killing the iteration the natural base form of that would be: yield raise to unset the existing exception and yeild it out instead, mirroring what a bare raise within an except: clause does.
On Sat, Nov 19, 2011 at 5:55 PM, Gregory P. Smith <greg@krypto.org> wrote:
Indeed, "yield raise" would be quite appropriate terminology for this new channel of communication. However, there's still a potential stack unwinding problem here. For asymmetric coroutines, we need to be able to communicate an arbitrary distance up the stack. This idea, as it stands, doesn't provide that - like an ordinary yielded value, it can only yield control one level out. So even if the innermost generator is left in a resumable state, any *external* iterators are still going to be terminated. It potentially becomes more useful in combination with 'yield from', since yielded exceptions would be passed up the stack, the same as yielded values. So, while it's a neat trick and very cool that it could be implemented with such a small change, I still don't see how it helps substantially with the challenge of allowing *any* Python frame on a coroutine stack, not just generator frames. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 2011-11-19 at 18:25 +1000, Nick Coghlan wrote:
I'll see if I can get that to work.
It's really no different than gen.throw.() only going one deep. It doesn't go all the way to the bottom "implicitly" and then unwinds back out.
Yes, I think it would work nicely with yield-from.
So, while it's a neat trick and very cool that it could be implemented with such a small change,
I think *any* needs to be qualified in some way. The frame inside need to have some mechanism to be suspended or to pass control to someplace else. (or through it) That mechanism needs to be explicitly spelled in some way. We really need a good example of what that would/should look like. What exactly is the behavior you are looking for? Cheers, Ron
On Sat, 2011-11-19 at 18:25 +1000, Nick Coghlan wrote:
I was thinking about this today and a few things occurred to me. You don't want to suspend a regular function. What would happen if another function tried to call it in it's suspended state? To make that work, each invocation would need it's own frame. That is why generators return a generator object instance. To do that with functions would require making a function call into a called-function-instance. To be able to suspend and resume that instance, you will need a reference to the instance rather than the function. So then you get something that is very much like a generator. The only difference is it doesn't have a yield. But in order for that to suspend, it would need some mechanism to suspend itself. Like a yield. And then we are right back to generators. The other approach is to use thread objects. Which create instances that can be suspended. The advantage of using threads, is that the thread manager can suspend and resume threads independently of what the thread is doing. But that requires more resources to do that and they may not be as efficient in tight loops. The two different approaches are completely separate. So I don't see how this effects either yield-from or a possible yield-raise feature. I'm probably missing something, but I can't put my finger on it. Cheers, Ron
On Sun, Nov 20, 2011 at 8:57 AM, Ron Adam <ron3200@gmail.com> wrote:
I'm probably missing something, but I can't put my finger on it.
Re-read the last discussion of Greg's coroutine PEP in the list archives. If every frame on the stack has to be a generator frame, you're effectively bifurcating Python into two languages - "normal Python" (which uses both functions and generators) and "coroutine Python" (which uses generators for everything). That's a bad idea, and the reason so many people prefer the thread-style model of greenlet based programming to the Twisted-style "inside out" model of event driven programming. This is a post where I highlight some of the issues with bifurcating the language, as well as the fact that "generators-all-the-way-down" *does* lead to bifurcation: http://mail.python.org/pipermail/python-ideas/2011-October/012570.html Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Nov 18, 2011 at 10:24 PM, Ron Adam <ron3200@gmail.com> wrote:
neat!
Open issues:
* A better name than 'throws' might be good.
I don't like adding another keyword or confusing things by adding the "throw" verb to a language that already firmly uses the verb "raise" when speaking of exceptions. A double word syntax might make sense here. It'd be good to keep 'yeild' in it to make it clear that this is offering the exception out in a non-terminal manner. yield raise MyException(x,y,z) or if you've caught an exception from something and want to pass that on to the caller without killing the iteration the natural base form of that would be: yield raise to unset the existing exception and yeild it out instead, mirroring what a bare raise within an except: clause does.
On Sat, Nov 19, 2011 at 5:55 PM, Gregory P. Smith <greg@krypto.org> wrote:
Indeed, "yield raise" would be quite appropriate terminology for this new channel of communication. However, there's still a potential stack unwinding problem here. For asymmetric coroutines, we need to be able to communicate an arbitrary distance up the stack. This idea, as it stands, doesn't provide that - like an ordinary yielded value, it can only yield control one level out. So even if the innermost generator is left in a resumable state, any *external* iterators are still going to be terminated. It potentially becomes more useful in combination with 'yield from', since yielded exceptions would be passed up the stack, the same as yielded values. So, while it's a neat trick and very cool that it could be implemented with such a small change, I still don't see how it helps substantially with the challenge of allowing *any* Python frame on a coroutine stack, not just generator frames. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 2011-11-19 at 18:25 +1000, Nick Coghlan wrote:
I'll see if I can get that to work.
It's really no different than gen.throw.() only going one deep. It doesn't go all the way to the bottom "implicitly" and then unwinds back out.
Yes, I think it would work nicely with yield-from.
So, while it's a neat trick and very cool that it could be implemented with such a small change,
I think *any* needs to be qualified in some way. The frame inside need to have some mechanism to be suspended or to pass control to someplace else. (or through it) That mechanism needs to be explicitly spelled in some way. We really need a good example of what that would/should look like. What exactly is the behavior you are looking for? Cheers, Ron
On Sat, 2011-11-19 at 18:25 +1000, Nick Coghlan wrote:
I was thinking about this today and a few things occurred to me. You don't want to suspend a regular function. What would happen if another function tried to call it in it's suspended state? To make that work, each invocation would need it's own frame. That is why generators return a generator object instance. To do that with functions would require making a function call into a called-function-instance. To be able to suspend and resume that instance, you will need a reference to the instance rather than the function. So then you get something that is very much like a generator. The only difference is it doesn't have a yield. But in order for that to suspend, it would need some mechanism to suspend itself. Like a yield. And then we are right back to generators. The other approach is to use thread objects. Which create instances that can be suspended. The advantage of using threads, is that the thread manager can suspend and resume threads independently of what the thread is doing. But that requires more resources to do that and they may not be as efficient in tight loops. The two different approaches are completely separate. So I don't see how this effects either yield-from or a possible yield-raise feature. I'm probably missing something, but I can't put my finger on it. Cheers, Ron
On Sun, Nov 20, 2011 at 8:57 AM, Ron Adam <ron3200@gmail.com> wrote:
I'm probably missing something, but I can't put my finger on it.
Re-read the last discussion of Greg's coroutine PEP in the list archives. If every frame on the stack has to be a generator frame, you're effectively bifurcating Python into two languages - "normal Python" (which uses both functions and generators) and "coroutine Python" (which uses generators for everything). That's a bad idea, and the reason so many people prefer the thread-style model of greenlet based programming to the Twisted-style "inside out" model of event driven programming. This is a post where I highlight some of the issues with bifurcating the language, as well as the fact that "generators-all-the-way-down" *does* lead to bifurcation: http://mail.python.org/pipermail/python-ideas/2011-October/012570.html Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
Gregory P. Smith -
Nick Coghlan -
Ron Adam