[Python-Dev] Problems with GeneratorExit deriving from Exception

[Chad Austin]

Hello Python-Dev,

Here at IMVU, we love Python 2.5's generators-as-coroutines.  That feature has
let us succinctly express algorithms that intermix asynchronous network requests
and UI operations without writing complicated state machines, and, perhaps most
importantly, get high-quality unit tests around these algorithms.

However, we've been having a problem with the way GeneratorExit interacts with
our coroutine system.  Let's take a bit of simplified example code from our system:

	@task
	def pollForChatInvites(chatGateway, userId):
		while True:
			try:
				# Network call.
				result = yield chatGateway.checkForInvite({'userId': userId})
			except Exception:  # An XML-RPC call can fail for many reasons.
				result = None
			# ... handle result here
			yield Sleep(10)

If a task (coroutine) is cancelled while it's waiting for the result from
checkForInvite, a GeneratorExit will be raised from that point in the generator,
which will be caught and ignored by the "except Exception:" clause, causing a
RuntimeError to be raised from whoever tried to close the generator.  Moreover,
any finally: clauses or with-statement contexts don't run.

We have also run into problems where a task tries to "return" (yield Return())
from within a try: except Exception: block.  Since returning from a coroutine is
roughly equivalent to "raise GeneratorExit", the exception can be caught and
ignored, with the same consequences as above.

This problem could be solved in several ways:

1) Make GeneratorExit derive from BaseException, just like SystemExit.

2) Add "except GeneratorExit: raise" to every usage of except Exception: in tasks.

3) Change the semantics of except clauses so that you can use any container as
an exception filter.  You could have a custom exception filter object that would
catch any Exception-derived exception except for GeneratorExit.  Then we'd have
to teach the team to use "except ImvuExceptionFilter:" rather than "except
Exception:".

I prefer option #1, because it matches SystemExit and I haven't ever seen a case
where I wanted to catch GeneratorExit.  When a generator is closed, I just want
finally: clauses to run, like a normal return statement would.  In fact, we have
already implemented option #1 locally, but would like it to be standard.

Option #2 would add needless noise throughout the system,

You could argue that it's bad style to catch Exception, but there are many
situations where that's exactly what I want.  I don't actually care _how_ the
xml-rpc call failed, just that the error is logged and the call is retried
later.  Same with loading caches from disk.

Proposals for changing GeneratorExit to be a BaseException have come up on this
list in the past [1] [2], but were rejected as being too "theoretical".  A
significant portion of the IMVU client is now specified with coroutines, so I
hope to resume this conversation.

Thoughts?

Chad

[1] http://mail.python.org/pipermail/python-dev/2006-March/062654.html
[2] http://mail.python.org/pipermail/python-dev/2006-March/062825.html

-- 
http://imvu.com/technology