On Fri, 2011-10-28 at 09:56 +1000, Nick Coghlan wrote:
Frankly, now that I understand the problem more clearly, attempting to attack it by making it easier to create stacks consisting entirely of generator frames strikes me as a terrible idea.
It seems to me, we can do better.
The 'yield' and 'yield from' statements are great for a single generator. And that's fine.
Once we start running multiples co-routines and start to switch between them, things start to get very complex.
A trampoline works by flattening out the stack so any sub-generator is always just below the trampoline runner. The trade off is that the trampoline runner then has to sort out how to handle what is yielded to it. That becomes additional overhead that then requires the sub-generators to also pass out signals to the trampoline runner or scheduler. It all goes through the same yield data paths, so it becomes additional overhead to sort that out.
In the tests I've done, once you get to that point, they start to run at about the same speed as using class's with a .run() method. No generators required to do that as the method call and return points correspond to one loop in a generator.
To avoid the additional overhead, a second suspend point that can yield out to a scheduler would be nice. It would avoid a lot of 'if-else's as the data yield path isn't mixed with trampoline and scheduler messages. I think there have been requests for "channels", but I'm not sure if that would solve these issues directly.
Far better to find a way to have a dynamic "non-local" yield construct that yields from the *outermost* generator frame in the stack, regardless of whether the current frame is a generator or not.
Named yields won't work of course, and you are probably not referring to the non_local keyword, but exceptions can have names, so non_local could be used get an exception defined in an outer scope. It's not unreasonable for exceptions to effect the control flow as StopIteration already does that. It's also not unreasonable for an Exception to only work in a few contexts.
Ideally, we would like to make it possible to write code like his:
def coroutine_friendly_io(*args, **kwds): if in_coroutine(): return coyield AsychronousRequest(async_op, *args, **kwds) else: return sync_op(*args, **kwds)
It's hard to tell just how this would work without seeing it's context. As I said above, once you start getting into more complex designs, it starts to require additional signals and checks of some sort as it appears you have done in this example.
Another issue with this is, the routines and the framework become tied together. The routines need to know the proper additional protocol to work with that framework, and they also can't be used with any other framework.
That probably isn't avoidable entirely. But if we take that into account at the start, we can probably make things much easier later.