[Python-ideas] Cofunctions - Getting away from the iterator protocol

[Ron Adam]

On Tue, 2011-11-01 at 18:15 +1000, Nick Coghlan wrote:
> On Tue, Nov 1, 2011 at 4:27 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> > I believe raise just instantiates the indicated exception. I expect that
> > Exception.__new__ or .__init__ captures the traceback info. Subclasses can
> > add more. A SuspendExecution exception should be able to grab as much as is
> > needed for a resume. A CAPI call could be added if needed.
> 
> No, the traceback info is added by the eval loop itself. Remember that
> when you raise an exception *type* (rather than an instance), the
> exception doesn't get instantiated until it gets caught somewhere -
> the eval loop maintains the unwinding stack for the traceback as part
> of the thread state until it is time to attach it to the exception
> object.
> 
> This is all at the tail end of the eval loop in CPython, but be warned
> it's fairly brain bending stuff that depends on various internal
> details of the eval loop:
> http://hg.python.org/cpython/file/default/Python/ceval.c#l2879

Thanks for the link, I've been trying to get my brain bent around it,
but, yes it is hard to understand how it all ties together.

This morning I had a thought and maybe it may lead somewhere... 


Would it be possible to rewrite the 'yield' internals so they work in
the following way...  

     # a = yield b
     try:
          raise SuspendException(b, _self=_self)
     Except ContinueException as exc:
          a = exc.args 


     # b = gen.send(a)
     def send(gen, a=None):
         try:
             gen.throw(ContinueException(a))
         except SuspendException as exc:
             (gen, *b) = exc.args
         return b


The two requirements for this to work are...

    *A SuspendException needs to be able to pass out of the
    generator without causing it to stop.

    *A throw needs to be able to work where the
    SuspendException was raised.


The next issue after that is how to allow a subclass of SuspendException
to get pass the next() or .send() caller.  A subclassed SuspendException
would still be caught by 'except SuspendException as exc'.  This is
needed as a scheduler or other outer framework sits outside the scope
the generator is *called in.  

*Exceptions work in the callers frame rather than the defining scope.
That's an important feature as it will allow coroutines much more
freedom to be used in different contexts.


What this does is give the non_local symantics you mentioned earlier. 

Cheers,
   Ron



> > I hope you keep looking at this idea. Function calls stop execution and pass
> > control 'down', to be resumed by return. yield stops execution and passes
> > control 'up', to be resumed by next (or .send). Exceptions pass control 'up'
> > (or 'out') without the possibility of resuming. All that is lacking is
> > something to suspend and pass control 'sideways', to a specific target. A
> > special exception makes some sense in that exceptions already get the call
> > stack needed to resume after suspension.
> 
> That's not actually true - due to the need to process exception
> handling clauses and finally blocks (including the implicit ones
> inside with statements), the internal state of those frames is
> potentially no longer valid for resumption (they've moved on beyond
> the point where the internal function was called).
> 
> I'll also note that it isn't necessary to pass control sideways, since
> there are two different flavours of coroutine design (the PDF article
> in the other thread describes this well). The Lua version is
> "asymmetric coroutines", and they only allow you to return to the
> point that first invoked the coroutine (this model is a fairly close
> fit with Python's generators and exception handling). The greenlet
> version is "symmetric" coroutines, and those let you switch directly
> to any other coroutine.
> 
> Both models have their pros and cons, but the main advantage of
> asymmetric coroutines is that you can just say "suspend this thread"
> without having to say *where* you want to switch to. Of course, you
> can implement much the same API with symmetric coroutines as well, so
> long as you can look up your parent coroutine easily. Ultimately, I
> expect the symmetric vs asymmetric decision will be driven more by
> implementation details than by philosophical preferences one way or
> the other.
> 
> I will note that Ron's suggestion to leverage the existing eval loop
> stack collection provided by the exception handling machinery does
> heavily favour the asymmetric approach. Having a quick look to refresh
> my memory of some of the details of CPython's exception handling, I've
> come to the following tentative conclusions:
> 
> - an ordinary exception won't do, since you don't want to trigger
> except and finally blocks in outer frames (ceval.c#2903)
> - in CPython, a new "why = WHY_SUSPEND" at the eval loop layer is
> likely a better approach, since it would allow the frame stack to be
> collected without triggering exception handling
> - the stack unwinding would then end when a "SETUP_COCALL" block was
> encountered on the block stack (just as SETUP_EXCEPT and SETUP_FINALLY
> can stop the stack unwinding following an exception
> - with the block stacks within the individual frames preserved, the
> collected stack should be in a fit state for later restoration
> - the "fast_yield" code and the generator resumption code should also
> provide useful insight
> 
> There's nothing too magical there - once we disclaim the ability to
> suspend coroutines while inside a C function (even one that has called
> back in via the C/Python API), it should boil down to a combination of
> the existing mechanics for generators and exception handling. So, even
> though the above description is (highly) CPython specific, it should
> be feasible for other implementations to come up with something
> similar (although perhaps not easy:
> http://lua-users.org/lists/lua-l/2007-07/msg00002.html).
> 
> Cheers,
> Nick.
>