[Python-ideas] Cofunctions - Back to Basics

[Nick Coghlan]

On Fri, Oct 28, 2011 at 8:10 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 28 October 2011 00:56, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> PEP 380 (i.e. "yield from") makes it easier to *create* those stacks
>> of generator frames, but it doesn't make the need for them to go away.
>> Proceeding further down that path (as PEP 3152 does) would essentially
>> partitioning Python programming into two distinct subsets: 'ordinary'
>> programming (where you can freely mix generators and ordinary function
>> frames) and 'generator coroutine' programming (where it *has* to be
>> generators all the way down to get suspension to work).
>
> This comment begs the question, is it the right thing to do to split
> Python programming into two subsets, as you suggest?

That's actually the status quo - you can right code normally, or you
can write it inside out Twisted style.

>> Frankly, now that I understand the problem more clearly, attempting to
>> attack it by making it easier to create stacks consisting entirely of
>> generator frames strikes me as a terrible idea. Far better to find a
>> way to have a dynamic "non-local" yield construct that yields from the
>> *outermost* generator frame in the stack, regardless of whether the
>> current frame is a generator or not.
>
> Hmm, OK. Sort of... I think the effect is correct, but it makes my
> head hurt having to think about it in terms of the internals of stacks
> and frames rather than in terms of my code... (I spend a few minutes
> thinking that the non-local yield should go *to* the generator frame,
> rather than yield *from* it. And doesn't this make generator loops
> within generator loops impossible? I don't now if they are needed, but
> I'd hate to bake in a limitation like that without considering if it's
> OK.)

Nah, the real beauty of a new mechanism is that it would leave the
existing generator channels untouched, so we wouldn't be running *any*
risk of breaking it. Generators would still use an ordinary yield and
borrow the stack frame of the caller - it's only the new coroutines
that would be creating a truly independent frame stack.

>> If you look at the greenlet docs
>> (http://packages.python.org/greenlet/) after reading the article on
>> Lua's coroutines, you'll realise that greenlet implements *symmetric*
>> coroutines - you have to explicitly state which greenlet you are
>> switching to. You can then implement asymmetric coroutines on top of
>> that by always switching to a specific scheduler thread.
>
> Given that the greenlet library exists (and is used by other projects,
> according to its PyPI page) why all the focus on core support?
> Seriously, how does the greenlet library fail to provide whatever
> users need? (Other than "it's not in the core" which could be fixed
> simply by adding it to the stdlib). I can't see any notes in the
> greenlet documentation which imply there are limitations/issues that
> might be a problem.

I believe greenlets supports switching even when there's a C function
on the stack, and they use hand written assembly code to do it.

So the answer to the question "Why not just use greenlets?" is the
following file (and its friends in that directory):
    https://bitbucket.org/ambroff/greenlet/src/77363116e78d/platform/switch_amd64_unix.h

That said, *without* that kind of assembly code, we'd probably face
the same problem Lua does (i.e. coroutines can't suspend when there's
a C function on the frame stack), which could potentially limit the
feature's usefulness (e.g. you migrate a function to C or Cython and
suddenly your coroutines break).

>> To achieve 'natural' coroutine programming, a Lua style asymmetric
>> coroutine approach looks the most promising to me.
>
> +1. Although I'm not sure whether this needs to be in the core (i.e.
> with language and/or syntax support), or in the stdlib, or just as a
> wrapper round the greenlet library.

I'm not sure either - I suspect that without the low level assembly
code that greenlets relies on a coroutine library for Python would
need support in the core to do the frame stack switching.

Years ago, I believe Guido was amenable to the idea of merging changes
back from Stackless, maybe he'd be open to the idea of supporting a
bit of assembly code in order to get full fledged coroutines in the
standard library. It has the usual activation barrier though (i.e.
someone seeking the blessing of the greenlets authors, writing a PEP
and championing it against all comers).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia