[Python-ideas] Cofunctions - Back to Basics

[Nick Coghlan]

On Thu, Oct 27, 2011 at 9:39 PM, Mark Shannon <mark at hotpy.org> wrote:
> Greg Ewing wrote:
>>
>> Mark Shannon wrote:
>>
>>> Why not have proper co-routines, instead of hacked-up generators?
>>
>> What do you mean by a "proper coroutine"?
>>
>
> A parallel, non-concurrent, thread of execution.
> It should be able to transfer control from arbitrary places in execution,
> not within generators.
>
> Stackless provides coroutines. Greenlets are also coroutines (I think).
>
> Lua has them, and is implemented in ANSI C, so it can be done portably.
> See: http://www.jucs.org/jucs_10_7/coroutines_in_lua/de_moura_a_l.pdf
>
> (One of the examples in the paper uses coroutines to implement generators,
> which is obviously not required in Python :) )

(It's kinda late here and I'm writing this without looking up details
in the source, so don't be shocked if some of the technical points
don't quite align with reality. The gist should still hold)

It's definitely an interesting point to think about. The thing that
makes generators fall short of being true coroutines is that they
don't really have the separate stacks that coroutines need. Instead,
they borrow the stack of whatever thread invoked next(), send() or
throw(). This means that, whenever a generator yields, that stack
needs to be unwound, suspending each affected generator in turn,
strung together by references between the generator objects rather
than remaining a true frame stack.

This means that *every* frame in the stack must be a generator frame
for the suspension to reach the outermost generator frame - ordinary
function frames can't be suspended like that.

So, let's suppose that instead of trying to change the way calls work
(to create generator frames all the way down), the coroutine PEP
instead proposed a new form of *yield* expression: coyield

The purpose of 'codef' would then be to declare that a function
maintains it's *own* stack frame, independent of that of the calling
thread. Unlike 'yield' (which only suspends the current frame),
'coyield' would instead suspend the entire coroutine, returning
control to the frame that invoked next(), send() or throw() on the
coroutine. Notably, *coyield* would *not* have any special effect on
the specific function that used it - instead, it would just be a
runtime error if coyield was encountered and there was no coroutine
frame on the stack.

That actually sounds reasonably feasible to me (and I like it better
than the "generator frames all the way down" approach). There are
likely to be some nasty implementation problems teasing out the thread
local state from the interpreter core though (and it poses interesting
problems for other things like the decimal module that also use thread
local state).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia