On Tue, 2011-11-01 at 18:15 +1000, Nick Coghlan wrote:
On Tue, Nov 1, 2011 at 4:27 PM, Terry Reedy email@example.com wrote:
I believe raise just instantiates the indicated exception. I expect that Exception.__new__ or .__init__ captures the traceback info. Subclasses can add more. A SuspendExecution exception should be able to grab as much as is needed for a resume. A CAPI call could be added if needed.
No, the traceback info is added by the eval loop itself. Remember that when you raise an exception *type* (rather than an instance), the exception doesn't get instantiated until it gets caught somewhere - the eval loop maintains the unwinding stack for the traceback as part of the thread state until it is time to attach it to the exception object.
This is all at the tail end of the eval loop in CPython, but be warned it's fairly brain bending stuff that depends on various internal details of the eval loop: http://hg.python.org/cpython/file/default/Python/ceval.c#l2879
Thanks for the link, I've been trying to get my brain bent around it, but, yes it is hard to understand how it all ties together.
This morning I had a thought and maybe it may lead somewhere...
Would it be possible to rewrite the 'yield' internals so they work in the following way...
# a = yield b try: raise SuspendException(b, _self=_self) Except ContinueException as exc: a = exc.args
# b = gen.send(a) def send(gen, a=None): try: gen.throw(ContinueException(a)) except SuspendException as exc: (gen, *b) = exc.args return b
The two requirements for this to work are...
*A SuspendException needs to be able to pass out of the generator without causing it to stop.
*A throw needs to be able to work where the SuspendException was raised.
The next issue after that is how to allow a subclass of SuspendException to get pass the next() or .send() caller. A subclassed SuspendException would still be caught by 'except SuspendException as exc'. This is needed as a scheduler or other outer framework sits outside the scope the generator is *called in.
*Exceptions work in the callers frame rather than the defining scope. That's an important feature as it will allow coroutines much more freedom to be used in different contexts.
What this does is give the non_local symantics you mentioned earlier.
I hope you keep looking at this idea. Function calls stop execution and pass control 'down', to be resumed by return. yield stops execution and passes control 'up', to be resumed by next (or .send). Exceptions pass control 'up' (or 'out') without the possibility of resuming. All that is lacking is something to suspend and pass control 'sideways', to a specific target. A special exception makes some sense in that exceptions already get the call stack needed to resume after suspension.
That's not actually true - due to the need to process exception handling clauses and finally blocks (including the implicit ones inside with statements), the internal state of those frames is potentially no longer valid for resumption (they've moved on beyond the point where the internal function was called).
I'll also note that it isn't necessary to pass control sideways, since there are two different flavours of coroutine design (the PDF article in the other thread describes this well). The Lua version is "asymmetric coroutines", and they only allow you to return to the point that first invoked the coroutine (this model is a fairly close fit with Python's generators and exception handling). The greenlet version is "symmetric" coroutines, and those let you switch directly to any other coroutine.
Both models have their pros and cons, but the main advantage of asymmetric coroutines is that you can just say "suspend this thread" without having to say *where* you want to switch to. Of course, you can implement much the same API with symmetric coroutines as well, so long as you can look up your parent coroutine easily. Ultimately, I expect the symmetric vs asymmetric decision will be driven more by implementation details than by philosophical preferences one way or the other.
I will note that Ron's suggestion to leverage the existing eval loop stack collection provided by the exception handling machinery does heavily favour the asymmetric approach. Having a quick look to refresh my memory of some of the details of CPython's exception handling, I've come to the following tentative conclusions:
- an ordinary exception won't do, since you don't want to trigger
except and finally blocks in outer frames (ceval.c#2903)
- in CPython, a new "why = WHY_SUSPEND" at the eval loop layer is
likely a better approach, since it would allow the frame stack to be collected without triggering exception handling
- the stack unwinding would then end when a "SETUP_COCALL" block was
encountered on the block stack (just as SETUP_EXCEPT and SETUP_FINALLY can stop the stack unwinding following an exception
- with the block stacks within the individual frames preserved, the
collected stack should be in a fit state for later restoration
- the "fast_yield" code and the generator resumption code should also
provide useful insight
There's nothing too magical there - once we disclaim the ability to suspend coroutines while inside a C function (even one that has called back in via the C/Python API), it should boil down to a combination of the existing mechanics for generators and exception handling. So, even though the above description is (highly) CPython specific, it should be feasible for other implementations to come up with something similar (although perhaps not easy: http://lua-users.org/lists/lua-l/2007-07/msg00002.html).