Problems with GeneratorExit deriving from Exception
Hello Python-Dev, Here at IMVU, we love Python 2.5's generators-as-coroutines. That feature has let us succinctly express algorithms that intermix asynchronous network requests and UI operations without writing complicated state machines, and, perhaps most importantly, get high-quality unit tests around these algorithms. However, we've been having a problem with the way GeneratorExit interacts with our coroutine system. Let's take a bit of simplified example code from our system: @task def pollForChatInvites(chatGateway, userId): while True: try: # Network call. result = yield chatGateway.checkForInvite({'userId': userId}) except Exception: # An XML-RPC call can fail for many reasons. result = None # ... handle result here yield Sleep(10) If a task (coroutine) is cancelled while it's waiting for the result from checkForInvite, a GeneratorExit will be raised from that point in the generator, which will be caught and ignored by the "except Exception:" clause, causing a RuntimeError to be raised from whoever tried to close the generator. Moreover, any finally: clauses or with-statement contexts don't run. We have also run into problems where a task tries to "return" (yield Return()) from within a try: except Exception: block. Since returning from a coroutine is roughly equivalent to "raise GeneratorExit", the exception can be caught and ignored, with the same consequences as above. This problem could be solved in several ways: 1) Make GeneratorExit derive from BaseException, just like SystemExit. 2) Add "except GeneratorExit: raise" to every usage of except Exception: in tasks. 3) Change the semantics of except clauses so that you can use any container as an exception filter. You could have a custom exception filter object that would catch any Exception-derived exception except for GeneratorExit. Then we'd have to teach the team to use "except ImvuExceptionFilter:" rather than "except Exception:". I prefer option #1, because it matches SystemExit and I haven't ever seen a case where I wanted to catch GeneratorExit. When a generator is closed, I just want finally: clauses to run, like a normal return statement would. In fact, we have already implemented option #1 locally, but would like it to be standard. Option #2 would add needless noise throughout the system, You could argue that it's bad style to catch Exception, but there are many situations where that's exactly what I want. I don't actually care _how_ the xml-rpc call failed, just that the error is logged and the call is retried later. Same with loading caches from disk. Proposals for changing GeneratorExit to be a BaseException have come up on this list in the past [1] [2], but were rejected as being too "theoretical". A significant portion of the IMVU client is now specified with coroutines, so I hope to resume this conversation. Thoughts? Chad [1] http://mail.python.org/pipermail/python-dev/2006-March/062654.html [2] http://mail.python.org/pipermail/python-dev/2006-March/062825.html -- http://imvu.com/technology
On Dec 1, 2007 2:38 PM, Chad Austin
Here at IMVU, we love Python 2.5's generators-as-coroutines. That feature has let us succinctly express algorithms that intermix asynchronous network requests and UI operations without writing complicated state machines, and, perhaps most importantly, get high-quality unit tests around these algorithms.
However, we've been having a problem with the way GeneratorExit interacts with our coroutine system. Let's take a bit of simplified example code from our system:
@task def pollForChatInvites(chatGateway, userId): while True: try: # Network call. result = yield chatGateway.checkForInvite({'userId': userId}) except Exception: # An XML-RPC call can fail for many reasons. result = None # ... handle result here yield Sleep(10)
If a task (coroutine) is cancelled while it's waiting for the result from checkForInvite, a GeneratorExit will be raised from that point in the generator, which will be caught and ignored by the "except Exception:" clause, causing a RuntimeError to be raised from whoever tried to close the generator. Moreover, any finally: clauses or with-statement contexts don't run.
We have also run into problems where a task tries to "return" (yield Return()) from within a try: except Exception: block. Since returning from a coroutine is roughly equivalent to "raise GeneratorExit", the exception can be caught and ignored, with the same consequences as above.
This problem could be solved in several ways:
1) Make GeneratorExit derive from BaseException, just like SystemExit.
2) Add "except GeneratorExit: raise" to every usage of except Exception: in tasks.
3) Change the semantics of except clauses so that you can use any container as an exception filter. You could have a custom exception filter object that would catch any Exception-derived exception except for GeneratorExit. Then we'd have to teach the team to use "except ImvuExceptionFilter:" rather than "except Exception:".
I prefer option #1, because it matches SystemExit and I haven't ever seen a case where I wanted to catch GeneratorExit. When a generator is closed, I just want finally: clauses to run, like a normal return statement would. In fact, we have already implemented option #1 locally, but would like it to be standard.
Option #2 would add needless noise throughout the system,
You could argue that it's bad style to catch Exception, but there are many situations where that's exactly what I want. I don't actually care _how_ the xml-rpc call failed, just that the error is logged and the call is retried later. Same with loading caches from disk.
Proposals for changing GeneratorExit to be a BaseException have come up on this list in the past [1] [2], but were rejected as being too "theoretical". A significant portion of the IMVU client is now specified with coroutines, so I hope to resume this conversation.
Thoughts?
Chad
[1] http://mail.python.org/pipermail/python-dev/2006-March/062654.html [2] http://mail.python.org/pipermail/python-dev/2006-March/062825.html
Well argued. I suggest to go for option (1) -- make GeneratorExit inherit from BaseException. We can do this starting 2.6. Feel free to upload a patch to bugs.python.org. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Dec 1, 2007 2:38 PM, Chad Austin
wrote: This problem could be solved in several ways:
1) Make GeneratorExit derive from BaseException, just like SystemExit.
Well argued. I suggest to go for option (1) -- make GeneratorExit inherit from BaseException. We can do this starting 2.6. Feel free to upload a patch to bugs.python.org.
Great! Patch is uploaded at http://bugs.python.org/issue1537 The patch changes the definition of GeneratorExit so that it extends BaseException, adds a generator test, updates exception_hierarchy.txt, and updates the exceptions page in the documentation. This is my first patch to Python -- did I miss anything? Thanks, Chad
On Dec 1, 2007 11:14 PM, Chad Austin
Guido van Rossum wrote:
On Dec 1, 2007 2:38 PM, Chad Austin
wrote: This problem could be solved in several ways:
1) Make GeneratorExit derive from BaseException, just like SystemExit.
Well argued. I suggest to go for option (1) -- make GeneratorExit inherit from BaseException. We can do this starting 2.6. Feel free to upload a patch to bugs.python.org.
Great! Patch is uploaded at http://bugs.python.org/issue1537
The patch changes the definition of GeneratorExit so that it extends BaseException, adds a generator test, updates exception_hierarchy.txt, and updates the exceptions page in the documentation. This is my first patch to Python -- did I miss anything?
I have not looked at the patch, so take what I say with a grain of salt. =) First, a generator test is not necessary. The patch changes the inheritance of exceptions, nothing more. While its usefulness is manifested for generators, this is really an exception detail. And changing exception_hierarchy.txt gives you the exception test you need. Second, the docs will need to be changed. I know that Doc/library/exceptions.rst needs a tweak. Not sure if anywhere else does. -Brett
Guido van Rossum wrote:
Well argued. I suggest to go for option (1) -- make GeneratorExit inherit from BaseException. We can do this starting 2.6. Feel free to upload a patch to bugs.python.org.
It actually took me a while to figure out why this use case was convincing, when the same idea had been rejected for 2.5. For anyone else as slow as me: in the hypothetical examples I posted before the release of 2.5, the yield could be moved to an else clause on the try-except statement without adversely affecting the semantics. The use case Chad presented here is different, because the exceptions to be handled are being passed back in via the yield expression - moving it would defeat the whole purpose of the exception handling. I'm sure the fact that the example comes from a real application rather than the 'what-if' generator in my brain helps a lot too :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
participants (4)
-
Brett Cannon
-
Chad Austin
-
Guido van Rossum
-
Nick Coghlan