On 28 October 2011 00:56, Nick Coghlan firstname.lastname@example.org wrote:
I highly recommend reading the article Mark Shannon linked earlier in the thread. I confess I didn't finish the whole thing, but even the first half of it made me realise *why* coroutine programming in Python (sans Stackless or greenlet) is such a pain: *every* frame on the coroutine stack has to be a generator frame in order to support suspending the generator.
Thanks for the link, and the detailed explanation. I will read the link, but as it's about Lua coroutines I am probably at least partially aware of the details, as I used to program a little in Lua, and understood coroutines from there (and other places, the idea is familiar to me).
I'm not against coroutines in principle (they certainly do have uses) but I wonder whether this PEP (and indeed PEP 342, where the basics of coroutines in Python came from) actually offer the sort of programming interface that people will actually use. To borrow further from Lua, they provide coroutines via a standard library module:
coroutine.create(fn) -- convert a function to a coroutine co.resume(arg...) -- resume a coroutine, with arguments coroutine.running() -- the current coroiutine co.status() -- running, suspended, normal or dead coroutine.wrap(f) -- helper combining create and resume coroutine.yield(arg...) -- yield back to whoever called resume
There's obviously language runtime support behind the scenes, but conceptually, this is the sort of model I think people can make real use of. It's precisely the sort of asymmetric model you mention later, and I agree that it's more user-friendly (albeit less powerful) than fully symmetric coroutines. So in my view, any coroutine PEP should be measured (at least in part) against how well it maps to these sorts of core concepts, at least if it's aimed at a non-specialist userbase.
PEP 380 (i.e. "yield from") makes it easier to *create* those stacks of generator frames, but it doesn't make the need for them to go away. Proceeding further down that path (as PEP 3152 does) would essentially partitioning Python programming into two distinct subsets: 'ordinary' programming (where you can freely mix generators and ordinary function frames) and 'generator coroutine' programming (where it *has* to be generators all the way down to get suspension to work).
This comment begs the question, is it the right thing to do to split Python programming into two subsets, as you suggest?
Frankly, now that I understand the problem more clearly, attempting to attack it by making it easier to create stacks consisting entirely of generator frames strikes me as a terrible idea. Far better to find a way to have a dynamic "non-local" yield construct that yields from the *outermost* generator frame in the stack, regardless of whether the current frame is a generator or not.
Hmm, OK. Sort of... I think the effect is correct, but it makes my head hurt having to think about it in terms of the internals of stacks and frames rather than in terms of my code... (I spend a few minutes thinking that the non-local yield should go *to* the generator frame, rather than yield *from* it. And doesn't this make generator loops within generator loops impossible? I don't now if they are needed, but I'd hate to bake in a limitation like that without considering if it's OK.)
Ideally, we would like to make it possible to write code like this:
def coroutine_friendly_io(*args, **kwds): if in_coroutine(): return coyield AsychronousRequest(async_op, *args, **kwds) else: return sync_op(*args, **kwds)
If you look at the greenlet docs (http://packages.python.org/greenlet/) after reading the article on Lua's coroutines, you'll realise that greenlet implements *symmetric* coroutines - you have to explicitly state which greenlet you are switching to. You can then implement asymmetric coroutines on top of that by always switching to a specific scheduler thread.
Given that the greenlet library exists (and is used by other projects, according to its PyPI page) why all the focus on core support? Seriously, how does the greenlet library fail to provide whatever users need? (Other than "it's not in the core" which could be fixed simply by adding it to the stdlib). I can't see any notes in the greenlet documentation which imply there are limitations/issues that might be a problem.
To achieve 'natural' coroutine programming, a Lua style asymmetric coroutine approach looks the most promising to me.
+1. Although I'm not sure whether this needs to be in the core (i.e. with language and/or syntax support), or in the stdlib, or just as a wrapper round the greenlet library.