[Python-ideas] Cofunctions - Back to Basics

[Ron Adam]

On Fri, 2011-10-28 at 09:56 +1000, Nick Coghlan wrote:
> Frankly, now that I understand the problem more clearly,
> attempting to attack it by making it easier to create stacks
> consisting entirely of generator frames strikes me as a
> terrible idea.

It seems to me, we can do better.

The  'yield' and 'yield from' statements are great for a single
generator.  And that's fine.

Once we start running multiples co-routines and start to switch between
them, things start to get very complex.

A trampoline works by flattening out the stack so any sub-generator is
always just below the trampoline runner. The trade off is that the
trampoline runner then has to sort out how to handle what is yielded to
it.  That becomes additional overhead that then requires the
sub-generators to also pass out signals to the trampoline runner or
scheduler.  It all goes through the same yield data paths, so it becomes
additional overhead to sort that out.

In the tests I've done, once you get to that point, they start to run at
about the same speed as using class's with a .run() method.  No
generators required to do that as the method call and return points
correspond to one loop in a generator.  

To avoid the additional overhead, a second suspend point that can yield
out to a scheduler would be nice. It would avoid a lot of 'if-else's as
the data yield path isn't mixed with trampoline and scheduler messages.
I think there have been requests for "channels", but I'm not sure if
that would solve these issues directly.


> Far better to find a
> way to have a dynamic "non-local" yield construct that yields from the
> *outermost* generator frame in the stack, regardless of whether the
> current frame is a generator or not.

Named yields won't work of course, and you are probably not referring to
the non_local keyword, but exceptions can have names, so non_local could
be used get an exception defined in an outer scope.  It's not
unreasonable for exceptions to effect the control flow as StopIteration
already does that.  It's also not unreasonable for an Exception to only
work in a few contexts.


> Ideally, we would like to make it possible to write code like his:
> 
>     def coroutine_friendly_io(*args, **kwds):
>         if in_coroutine():
>             return coyield AsychronousRequest(async_op, *args, **kwds)
>         else:
>             return sync_op(*args, **kwds) 

It's hard to tell just how this would work without seeing it's context.
As I said above, once you start getting into more complex designs, it
starts to require additional signals and checks of some sort as it
appears you have done in this example.

Another issue with this is, the routines and the framework become tied
together.  The routines need to know the proper additional protocol to
work with that framework, and they also can't be used with any other
framework.

That probably isn't avoidable entirely.  But if we take that into
account at the start, we can probably make things much easier later.

Cheers,
   Ron