[Python-ideas] Coroutines and PEP 380

[Matt Joiner]

On Thu, Jan 26, 2012 at 5:47 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> My understanding (in general, and specifically of the various
> discussions that have happened on python-ideas) is that we're talking
> about co-operative coroutines.
>
> To be clear, what I mean by that is that a coroutine switch is invoked
> by an explicit call/statement (which may be "yield from" or some
> as-yet unspecified variation - a cocall statement, a call to a
> switch() method on a coroutine object, or whatever). The discussions
> seem to be more about whether incrementally extending generators (with
> send methods, yield from, etc) to turn them into coroutines is better
> or worse than starting with a new approach which is closer to
> "conventional" approaches from other languages with first-class
> coroutines (for example Lua, which uses method calls on coroutine
> objects).
>
> I presume that "preemptive coroutines" means that a coroutine switch
> can occur at any point. I can't see how that is in any practical sense
> different from a thread...

I'm not sure that preemptive coroutines as you've described them are
what people are aiming for.

Without better terminology I've taken to calling them implicit and
explicit coroutines:

Implicit coroutines can arbitrary switch without changes in syntax,
calling convention, or traversing back up the stack.

Explicit coroutines must be "driven" (yield from), are called
differently (yield from), and send values back up the stack before
they can switch cleanly (yield from).

Clearly all coroutines share several fantastic properties, they:

* Are stored on the heap,
* Are not preempted,
* They can execute within a single native thread,
* Store state without tying up a native stack for this purpose.

Used for concurrency with an eventing scheduler in Python they give:

* Massive concurrency with minimal system impact (try 20k threads
instead of 20k coroutines),
* Side-step the GIL,
* Ridiculously cheap context switching,
* Remove the need for callbacks and all that rubbish (I saw Guido was
keen on this one),
* Allow existing synchronous code to remain unchanged by modifying the
underlying blocking calls (implicit coroutines only).