Re: [Python-ideas] Coroutines and PEP 380

(redirecting to python-ideas - coroutine proposals are nowhere near mature enough for python-dev) On Wed, Jan 25, 2012 at 5:35 PM, Matt Joiner <anacrolix@gmail.com> wrote:
If someone can explain what's stopping real coroutines being into Python (3.3), that would be great.
The general issues with that kind of idea: - the author hasn't offered the code for inclusion and relicensing under the PSF license (thus we legally aren't allowed to do it) - complexity - maintainability - platform support In the specific case of coroutines, you have the additional hurdle of convincing people whether or not they're a good idea at all. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Jan 25, 2012 at 3:45 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think coroutines can be immensely useful. I am using continulets in PyPy to implement the flow control for symbolic expression capture, and it is by far the least complex method I have examined. I can't think of any other examples where I've really felt blocked by a standard stack though. Nathan

On Wed, 25 Jan 2012 18:45:16 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Do you mean preemptive coroutines as opposed to ("yield from"-based) cooperative coroutines? Because I don't see the point of PEP 380 if not for coroutines (tree-walking examples are academic). (I'd also like to point out the following paper which makes an useful distinction between "task management" and "stack management": http://www.usenix.org/event/usenix02/adyahowell.html ) Regards Antoine.

On 25 January 2012 14:45, Antoine Pitrou <solipsis@pitrou.net> wrote:
Do you mean preemptive coroutines as opposed to ("yield from"-based) cooperative coroutines?
My understanding (in general, and specifically of the various discussions that have happened on python-ideas) is that we're talking about co-operative coroutines. To be clear, what I mean by that is that a coroutine switch is invoked by an explicit call/statement (which may be "yield from" or some as-yet unspecified variation - a cocall statement, a call to a switch() method on a coroutine object, or whatever). The discussions seem to be more about whether incrementally extending generators (with send methods, yield from, etc) to turn them into coroutines is better or worse than starting with a new approach which is closer to "conventional" approaches from other languages with first-class coroutines (for example Lua, which uses method calls on coroutine objects). I presume that "preemptive coroutines" means that a coroutine switch can occur at any point. I can't see how that is in any practical sense different from a thread... Paul.

On Thu, Jan 26, 2012 at 5:47 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'm not sure that preemptive coroutines as you've described them are what people are aiming for. Without better terminology I've taken to calling them implicit and explicit coroutines: Implicit coroutines can arbitrary switch without changes in syntax, calling convention, or traversing back up the stack. Explicit coroutines must be "driven" (yield from), are called differently (yield from), and send values back up the stack before they can switch cleanly (yield from). Clearly all coroutines share several fantastic properties, they: * Are stored on the heap, * Are not preempted, * They can execute within a single native thread, * Store state without tying up a native stack for this purpose. Used for concurrency with an eventing scheduler in Python they give: * Massive concurrency with minimal system impact (try 20k threads instead of 20k coroutines), * Side-step the GIL, * Ridiculously cheap context switching, * Remove the need for callbacks and all that rubbish (I saw Guido was keen on this one), * Allow existing synchronous code to remain unchanged by modifying the underlying blocking calls (implicit coroutines only).

On 26 January 2012 02:08, Matt Joiner <anacrolix@gmail.com> wrote:
That's what confuses me. If an implicit coroutine can arbitrarily switch without changes in syntax, etc, how is it not pre-empted? How does the switch happen, if not by pre-emption or by some explicit request/syntax? An example of what you mean would probably help. Paul

Le 26/01/2012 03:08, Matt Joiner a écrit :
Implicit coroutines can arbitrary switch without changes in syntax, calling convention, or traversing back up the stack.
Do you mean that greenlets? some_greenlet.switch() looks like a method call and does not required a special syntax like "yield from" so it can be hidden in some_file.read() like gevent does. Regards, -- Simon Sapin

On Wed, Jan 25, 2012 at 3:45 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think coroutines can be immensely useful. I am using continulets in PyPy to implement the flow control for symbolic expression capture, and it is by far the least complex method I have examined. I can't think of any other examples where I've really felt blocked by a standard stack though. Nathan

On Wed, 25 Jan 2012 18:45:16 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Do you mean preemptive coroutines as opposed to ("yield from"-based) cooperative coroutines? Because I don't see the point of PEP 380 if not for coroutines (tree-walking examples are academic). (I'd also like to point out the following paper which makes an useful distinction between "task management" and "stack management": http://www.usenix.org/event/usenix02/adyahowell.html ) Regards Antoine.

On 25 January 2012 14:45, Antoine Pitrou <solipsis@pitrou.net> wrote:
Do you mean preemptive coroutines as opposed to ("yield from"-based) cooperative coroutines?
My understanding (in general, and specifically of the various discussions that have happened on python-ideas) is that we're talking about co-operative coroutines. To be clear, what I mean by that is that a coroutine switch is invoked by an explicit call/statement (which may be "yield from" or some as-yet unspecified variation - a cocall statement, a call to a switch() method on a coroutine object, or whatever). The discussions seem to be more about whether incrementally extending generators (with send methods, yield from, etc) to turn them into coroutines is better or worse than starting with a new approach which is closer to "conventional" approaches from other languages with first-class coroutines (for example Lua, which uses method calls on coroutine objects). I presume that "preemptive coroutines" means that a coroutine switch can occur at any point. I can't see how that is in any practical sense different from a thread... Paul.

On Thu, Jan 26, 2012 at 5:47 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'm not sure that preemptive coroutines as you've described them are what people are aiming for. Without better terminology I've taken to calling them implicit and explicit coroutines: Implicit coroutines can arbitrary switch without changes in syntax, calling convention, or traversing back up the stack. Explicit coroutines must be "driven" (yield from), are called differently (yield from), and send values back up the stack before they can switch cleanly (yield from). Clearly all coroutines share several fantastic properties, they: * Are stored on the heap, * Are not preempted, * They can execute within a single native thread, * Store state without tying up a native stack for this purpose. Used for concurrency with an eventing scheduler in Python they give: * Massive concurrency with minimal system impact (try 20k threads instead of 20k coroutines), * Side-step the GIL, * Ridiculously cheap context switching, * Remove the need for callbacks and all that rubbish (I saw Guido was keen on this one), * Allow existing synchronous code to remain unchanged by modifying the underlying blocking calls (implicit coroutines only).

On 26 January 2012 02:08, Matt Joiner <anacrolix@gmail.com> wrote:
That's what confuses me. If an implicit coroutine can arbitrarily switch without changes in syntax, etc, how is it not pre-empted? How does the switch happen, if not by pre-emption or by some explicit request/syntax? An example of what you mean would probably help. Paul

Le 26/01/2012 03:08, Matt Joiner a écrit :
Implicit coroutines can arbitrary switch without changes in syntax, calling convention, or traversing back up the stack.
Do you mean that greenlets? some_greenlet.switch() looks like a method call and does not required a special syntax like "yield from" so it can be hidden in some_file.read() like gevent does. Regards, -- Simon Sapin
participants (7)
-
Antoine Pitrou
-
Matt Joiner
-
Nathan Rice
-
Nick Coghlan
-
Paul Moore
-
Simon Sapin
-
Yuval Greenfield