[Python-Dev] PEP 342 suggestion: start(), __call__() and unwind_call() methods

[Nick Coghlan]

I'm lifting Jason's PEP 342 suggestions out of the recent PEP 343 thread, in 
case some of the folks interested in coroutines stopped following that discussion.

Jason suggested two convenience methods, .start() and .finish().

start() simply asserted that the generator hadn't been started yet, and I find 
the parallel with "Thread.start()" appealing:

         def start(self):
             """ Convenience method -- exactly like next(), but
             assert that this coroutine hasn't already been started.
             """
             if self.__started:
                 raise RuntimeError("Coroutine already started")
             return self.next()


I've embellished Jason's suggested finish() method quite a bit though.
   1. Use send() rather than next()
   2. Call it __call__() rather than finish()
   3. Add an unwind_call() variant that gives similar semantics for throw()
   4. Support getting a return value from the coroutine
      using the syntax "raise StopIteration(val)"
   5. Add an exception "ContinueIteration" that is used to indicate the
      generator hasn't finished yet, rather than expecting the generator to
      finish and raising RuntimeError if it doesn't

It ends up looking like this:

         def __call__(self, value=None):
             """ Call a generator as a coroutine

             Returns the first argument supplied to StopIteration or
             None if no argument was supplied.
             Raises ContinueIteration with the value yielded as the
             argument if the generator yields a value
             """
             if not self.__started:
                 raise RuntimeError("Coroutine not started")
             try:
                 if exc:
                     yield_val = self.throw(value, *exc)
                 else:
                     yield_val = self.send(value)
             except (StopIteration), ex:
                 if ex.args:
                     return args[0]
             else:
                 raise ContinueIteration(yield_val)

         def unwind_call(self, *exc):
             """Raise an exception in a generator used as a coroutine.

             Returns the first argument supplied to StopIteration or
             None if no argument was supplied.
             Raises ContinueIteration if the generator yields a value
             with the value yield as the argument
             """
             try:
                 yield_val = self.throw(*exc)
             except (StopIteration), ex:
                 if ex.args:
                     return args[0]
             else:
                 raise ContinueIteration(yield_val)

Now here's the trampoline scheduler from PEP 342 using this idea:

         import collections

         class Trampoline:
             """Manage communications between coroutines"""

             running = False

             def __init__(self):
                 self.queue = collections.deque()

             def add(self, coroutine):
                 """Request that a coroutine be executed"""
                 self.schedule(coroutine)

             def run(self):
                 result = None
                 self.running = True
                 try:
                     while self.running and self.queue:
                         func = self.queue.popleft()
                         result = func()
                     return result
                 finally:
                     self.running = False

             def stop(self):
                 self.running = False

             def schedule(self, coroutine, stack=(), call_result=None, *exc):
                 # Define the new pseudothread
                 def pseudothread():
                     try:
                         if exc:
                             result = coroutine.unwind_call(call_result, *exc)
                         else:
                             result = coroutine(call_result)
                     except (ContinueIteration), ex:
                         # Called another coroutine
                         callee = ex.args[0]
                         self.schedule(callee, (coroutine,stack))
                     except:
                         if stack:
                             # send the error back to the caller
                            caller = stack[0]
                            prev_stack = stack[1]
                            self.schedule(
                                 caller, prev_stack, *sys.exc_info()
                            )
                         else:
                             # Nothing left in this pseudothread to
                             # handle it, let it propagate to the
                             # run loop
                             raise
                     else:
                         if stack:
                             # Finished, so pop the stack and send the
                             # result to the caller
                             caller = stack[0]
                             prev_stack = stack[1]
                             self.schedule(caller, prev_stack, result)

                 # Add the new pseudothread to the execution queue
                 self.queue.append(pseudothread)

Notice how a non-coroutine callable can be yielded, and it will still work 
happily with the scheduler, because the desire to continue execution is 
indicated by the ContinueIteration exception, rather than by the type of the 
returned value.

With this relatively dumb scheduler, that doesn't provide any particular 
benefit - the specific pseudothread doesn't block, but eventually the 
scheduler itself blocks when it executes the non-coroutine call.

However, it wouldn't take too much to make the scheduler smarter and give it a 
physical thread pool that it used whenever it encountered a non-coroutine call

And that's the real trick here: with these additions to PEP 342, the decision 
of how to deal with blocking calls could be made in the scheduler, without 
affecting the individual coroutines. All the coroutine writers would need to 
remember to do is to write any potentially blocking operations as yielded 
lambda expressions or functional.partial invocations, rather than as direct 
function calls.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.blogspot.com