[Python-ideas] Cofunctions - Back to Basics

[Paul Moore]

On 28 October 2011 00:56, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I highly recommend reading the article Mark Shannon linked earlier in
> the thread. I confess I didn't finish the whole thing, but even the
> first half of it made me realise *why* coroutine programming in Python
> (sans Stackless or greenlet) is such a pain: *every* frame on the
> coroutine stack has to be a generator frame in order to support
> suspending the generator.

Thanks for the link, and the detailed explanation. I will read the
link, but as it's about Lua coroutines I am probably at least
partially aware of the details, as I used to program a little in Lua,
and understood coroutines from there (and other places, the idea is
familiar to me).

I'm not against coroutines in principle (they certainly do have uses)
but I wonder whether this PEP (and indeed PEP 342, where the basics of
coroutines in Python came from) actually offer the sort of programming
interface that people will actually use. To borrow further from Lua,
they provide coroutines via a standard library module:

coroutine.create(fn) -- convert a function to a coroutine
co.resume(arg...) -- resume a coroutine, with arguments
coroutine.running() -- the current coroiutine
co.status() -- running, suspended, normal or dead
coroutine.wrap(f) -- helper combining create and resume
coroutine.yield(arg...) -- yield back to whoever called resume

There's obviously language runtime support behind the scenes, but
conceptually, this is the sort of model I think people can make real
use of. It's precisely the sort of asymmetric model you mention later,
and I agree that it's more user-friendly (albeit less powerful) than
fully symmetric coroutines. So in my view, any coroutine PEP should be
measured (at least in part) against how well it maps to these sorts of
core concepts, at least if it's aimed at a non-specialist userbase.

> PEP 380 (i.e. "yield from") makes it easier to *create* those stacks
> of generator frames, but it doesn't make the need for them to go away.
> Proceeding further down that path (as PEP 3152 does) would essentially
> partitioning Python programming into two distinct subsets: 'ordinary'
> programming (where you can freely mix generators and ordinary function
> frames) and 'generator coroutine' programming (where it *has* to be
> generators all the way down to get suspension to work).

This comment begs the question, is it the right thing to do to split
Python programming into two subsets, as you suggest?

> Frankly, now that I understand the problem more clearly, attempting to
> attack it by making it easier to create stacks consisting entirely of
> generator frames strikes me as a terrible idea. Far better to find a
> way to have a dynamic "non-local" yield construct that yields from the
> *outermost* generator frame in the stack, regardless of whether the
> current frame is a generator or not.

Hmm, OK. Sort of... I think the effect is correct, but it makes my
head hurt having to think about it in terms of the internals of stacks
and frames rather than in terms of my code... (I spend a few minutes
thinking that the non-local yield should go *to* the generator frame,
rather than yield *from* it. And doesn't this make generator loops
within generator loops impossible? I don't now if they are needed, but
I'd hate to bake in a limitation like that without considering if it's
OK.)

> Ideally, we would like to make it possible to write code like this:
>
>    def coroutine_friendly_io(*args, **kwds):
>        if in_coroutine():
>            return coyield AsychronousRequest(async_op, *args, **kwds)
>        else:
>            return sync_op(*args, **kwds)
>
> If you look at the greenlet docs
> (http://packages.python.org/greenlet/) after reading the article on
> Lua's coroutines, you'll realise that greenlet implements *symmetric*
> coroutines - you have to explicitly state which greenlet you are
> switching to. You can then implement asymmetric coroutines on top of
> that by always switching to a specific scheduler thread.

Given that the greenlet library exists (and is used by other projects,
according to its PyPI page) why all the focus on core support?
Seriously, how does the greenlet library fail to provide whatever
users need? (Other than "it's not in the core" which could be fixed
simply by adding it to the stdlib). I can't see any notes in the
greenlet documentation which imply there are limitations/issues that
might be a problem.

> To achieve 'natural' coroutine programming, a Lua style asymmetric
> coroutine approach looks the most promising to me.

+1. Although I'm not sure whether this needs to be in the core (i.e.
with language and/or syntax support), or in the stdlib, or just as a
wrapper round the greenlet library.

Paul