Re: [Python-ideas] Cofunctions - Back to Basics
Greg Ewing wrote:
Mark Shannon wrote:
Why not have proper co-routines, instead of hacked-up generators?
What do you mean by a "proper coroutine"?
A parallel, non-concurrent, thread of execution. It should be able to transfer control from arbitrary places in execution, not within generators. Stackless provides coroutines. Greenlets are also coroutines (I think). Lua has them, and is implemented in ANSI C, so it can be done portably. See: http://www.jucs.org/jucs_10_7/coroutines_in_lua/de_moura_a_l.pdf (One of the examples in the paper uses coroutines to implement generators, which is obviously not required in Python :) ) Cheers, Mark.
On Thu, Oct 27, 2011 at 9:39 PM, Mark Shannon <mark@hotpy.org> wrote:
Greg Ewing wrote:
Mark Shannon wrote:
Why not have proper co-routines, instead of hacked-up generators?
What do you mean by a "proper coroutine"?
A parallel, non-concurrent, thread of execution. It should be able to transfer control from arbitrary places in execution, not within generators.
Stackless provides coroutines. Greenlets are also coroutines (I think).
Lua has them, and is implemented in ANSI C, so it can be done portably. See: http://www.jucs.org/jucs_10_7/coroutines_in_lua/de_moura_a_l.pdf
(One of the examples in the paper uses coroutines to implement generators, which is obviously not required in Python :) )
(It's kinda late here and I'm writing this without looking up details in the source, so don't be shocked if some of the technical points don't quite align with reality. The gist should still hold) It's definitely an interesting point to think about. The thing that makes generators fall short of being true coroutines is that they don't really have the separate stacks that coroutines need. Instead, they borrow the stack of whatever thread invoked next(), send() or throw(). This means that, whenever a generator yields, that stack needs to be unwound, suspending each affected generator in turn, strung together by references between the generator objects rather than remaining a true frame stack. This means that *every* frame in the stack must be a generator frame for the suspension to reach the outermost generator frame - ordinary function frames can't be suspended like that. So, let's suppose that instead of trying to change the way calls work (to create generator frames all the way down), the coroutine PEP instead proposed a new form of *yield* expression: coyield The purpose of 'codef' would then be to declare that a function maintains it's *own* stack frame, independent of that of the calling thread. Unlike 'yield' (which only suspends the current frame), 'coyield' would instead suspend the entire coroutine, returning control to the frame that invoked next(), send() or throw() on the coroutine. Notably, *coyield* would *not* have any special effect on the specific function that used it - instead, it would just be a runtime error if coyield was encountered and there was no coroutine frame on the stack. That actually sounds reasonably feasible to me (and I like it better than the "generator frames all the way down" approach). There are likely to be some nasty implementation problems teasing out the thread local state from the interpreter core though (and it poses interesting problems for other things like the decimal module that also use thread local state). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, 2011-10-27 at 22:25 +1000, Nick Coghlan wrote:
On Thu, Oct 27, 2011 at 9:39 PM, Mark Shannon <mark@hotpy.org> wrote:
Greg Ewing wrote:
Mark Shannon wrote:
Why not have proper co-routines, instead of hacked-up generators?
What do you mean by a "proper coroutine"?
A parallel, non-concurrent, thread of execution. It should be able to transfer control from arbitrary places in execution, not within generators.
Stackless provides coroutines. Greenlets are also coroutines (I think).
Lua has them, and is implemented in ANSI C, so it can be done portably. See: http://www.jucs.org/jucs_10_7/coroutines_in_lua/de_moura_a_l.pdf
(One of the examples in the paper uses coroutines to implement generators, which is obviously not required in Python :) )
(It's kinda late here and I'm writing this without looking up details in the source, so don't be shocked if some of the technical points don't quite align with reality. The gist should still hold)
It's definitely an interesting point to think about. The thing that makes generators fall short of being true coroutines is that they don't really have the separate stacks that coroutines need. Instead, they borrow the stack of whatever thread invoked next(), send() or throw(). This means that, whenever a generator yields, that stack needs to be unwound, suspending each affected generator in turn, strung together by references between the generator objects rather than remaining a true frame stack.
This means that *every* frame in the stack must be a generator frame for the suspension to reach the outermost generator frame - ordinary function frames can't be suspended like that.
So, let's suppose that instead of trying to change the way calls work (to create generator frames all the way down), the coroutine PEP instead proposed a new form of *yield* expression: coyield
The purpose of 'codef' would then be to declare that a function maintains it's *own* stack frame, independent of that of the calling thread. Unlike 'yield' (which only suspends the current frame), 'coyield' would instead suspend the entire coroutine, returning control to the frame that invoked next(), send() or throw() on the coroutine. Notably, *coyield* would *not* have any special effect on the specific function that used it - instead, it would just be a runtime error if coyield was encountered and there was no coroutine frame on the stack.
That actually sounds reasonably feasible to me (and I like it better than the "generator frames all the way down" approach). There are likely to be some nasty implementation problems teasing out the thread local state from the interpreter core though (and it poses interesting problems for other things like the decimal module that also use thread local state).
Currently the only way to suspend a generator is at a 'yield' statement. (as far as I know) Would it be possible to allow a "raise SuspendIteration" exception to suspend a generator that could then be continued by calling a __resume__ method on the caught SuspendIteration exception? (It might use the traceback info to restart the generator in that case.) Maybe the scheduler could do something like... for gen in generators: try: # Continue a suspended generator gen._stored_suspenditeration.__resume__() except SuspendIteration as exc: # Continued generator stopped, so save the exception. gen._stored_suspenditeration = exc except StopIteration: generators.remove(gen) This might use the existing stack frame to do the work. I'm thinking if the scheduler could stay out of the way of the yield data path, it may make things easier. Cheers, Ron
Hi, Full coroutines have a number of advantages over the proposed cofunctions. For one, coroutines require *no* language changes. Here is a simple coroutine class interface (feel free to substitute/add your own names/signatures). class Coroutine: def __init__(self, callable): ... def resume(value): '''Passes value to coroutine, either as initial parameter to callable or as return value of yield''' def co_yield(value): 'Yields (returns) value back to caller of resume() method.' def stop(): 'Stops the coroutine' In order to implement this, it must be possible to transfer control *sideways* from one frame to another, without unwinding either stack. In order to do this, Python-to-Python calls within the interpreter must not make calls at the C (OS/hardware) level. This is quite simple to implement in practice. Currently CPython does something like this (for Python to Python calls). In interpreter: call PyFunction.__call__(parameters) In PyFunction.__call__(): Create new frame with parameters. Call interpreter with new frame. By changing the call sequence to something like: In interpreter: frame = PyFunction.make_frame(parameters) Push frame to frame-stack jump to start of new function. we have a 'stackless' implementation which can support coroutines. The implementation of Coroutines in a portable fashion is explained on pages 7 and 8 of this paper: http://www.lua.org/doc/jucs05.pdf (Sorry to keep quoting Lua papers, but it is a cleaner VM than CPython. Python is a better language, though :) ) Stackless uses a slightly different approach, not for any fundamental reason, but because their goal is to minimise the diff with CPython. Adding coroutines may create problems for the other implementations. PyPy already has a stackless implementation, but Jython and IronPython might find to it harder to implement. Any Jython or IronPython folks care to comment? In summary, the cofunction proposal is a work-around for a limitation in the VM. By fixing the VM we can have proper coroutines. Surely, it is better to make the VM support the features we want/need rather than bend those features to fit the VM? Cheers, Mark.
Errata to previous email.
def co_yield(value): 'Yields (returns) value back to caller of resume() method.'
Should have been @staticmethod def co_yield(value): 'Yields (returns) value back to caller of resume() method.' Cheers, Mark.
I'm sorry, I still don't understand what the problem here is. I didn't have any trouble making a python implementation for the wikipedia coroutine example: http://codepad.org/rEgg4GzW On Fri, Oct 28, 2011 at 12:16 PM, Mark Shannon <mark@hotpy.org> wrote:
Errata to previous email.
def co_yield(value): 'Yields (returns) value back to caller of resume() method.'
Should have been
@staticmethod
def co_yield(value): 'Yields (returns) value back to caller of resume() method.'
Cheers, Mark. ______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
Yuval Greenfield wrote:
I'm sorry, I still don't understand what the problem here is. I didn't have any trouble making a python implementation for the wikipedia coroutine example:
Where is the yield method? You need to be able to call Coroutine.co_yield() from *anywhere* not just in a generator. Suppose we have a Tree class with a walk method which takes a callback function: class Tree: class Node: def visit(self, callback): if self.left: self.left.visit(callback) if self.right: self.right.visit(callback) callback(self.value) def walk(self, callback): '''Recursively walks the tree calling callback(node) for each node''' self.root.visit(callback) We can then use Coroutine to create a tree iterator, with something like: def tree_callback(node): Coroutine.co_yield(node) #Make an iterator from a coroutine def tree_iterator(tree): co = Coroutine(tree.walk) yield co.resume(tree_callback) while True: yield co.resume(None) (I am glossing over questions like: Should a stopping coroutine raise an exception from the resume method or just return a value. Should a stopped coroutine that is resumed raise a StopIteration exception, a GeneratorExit exception or some new exception, etc, etc...) The important point here is that Tree.walk() is recursive and knows nothing of generators or coroutines, yet can be made to drive a generator.
On Fri, Oct 28, 2011 at 12:16 PM, Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org>> wrote:
Errata to previous email.
> > def co_yield(value): > 'Yields (returns) value back to caller of resume() method.'
Should have been
@staticmethod
def co_yield(value): 'Yields (returns) value back to caller of resume() method.'
Cheers, Mark. _________________________________________________ Python-ideas mailing list Python-ideas@python.org <mailto:Python-ideas@python.org> http://mail.python.org/__mailman/listinfo/python-ideas <http://mail.python.org/mailman/listinfo/python-ideas>
Mark Shannon wrote:
Stackless provides coroutines. Greenlets are also coroutines (I think).
Lua has them, and is implemented in ANSI C, so it can be done portably.
These all have drawbacks. Greenlets are based on non-portable (and, I believe, slightly dangerous) C hackery, and I'm given to understand that Lua coroutines can't be suspended from within a C function. My proposal has limitations, but it has the advantage of being based on fully portable and well-understood techniques. -- Greg
Greg Ewing wrote:
Mark Shannon wrote:
Stackless provides coroutines. Greenlets are also coroutines (I think).
Lua has them, and is implemented in ANSI C, so it can be done portably.
These all have drawbacks. Greenlets are based on non-portable (and, I believe, slightly dangerous) C hackery, and I'm given to understand that Lua coroutines can't be suspended from within a C function.
My proposal has limitations, but it has the advantage of being based on fully portable and well-understood techniques.
If Stackless has them, could we use that code? ~Ethan~
On Sat, Oct 29, 2011 at 8:40 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Greg Ewing wrote:
Mark Shannon wrote:
Stackless provides coroutines. Greenlets are also coroutines (I think).
Lua has them, and is implemented in ANSI C, so it can be done portably.
These all have drawbacks. Greenlets are based on non-portable (and, I believe, slightly dangerous) C hackery, and I'm given to understand that Lua coroutines can't be suspended from within a C function.
My proposal has limitations, but it has the advantage of being based on fully portable and well-understood techniques.
If Stackless has them, could we use that code?
That's what the greenlets module *is* - the coroutine code from Stackless, lifted out and provided as an extension module instead of a forked version of the runtime. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Oct 29, 2011 at 8:15 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Mark Shannon wrote:
Stackless provides coroutines. Greenlets are also coroutines (I think).
Lua has them, and is implemented in ANSI C, so it can be done portably.
These all have drawbacks. Greenlets are based on non-portable (and, I believe, slightly dangerous) C hackery, and I'm given to understand that Lua coroutines can't be suspended from within a C function.
My proposal has limitations, but it has the advantage of being based on fully portable and well-understood techniques.
The limitation of Lua style coroutines is that they can't be suspended from inside a function implemented in C. Without greenlets/Stackless style assembly code, coroutines in Python would likely have the same limitation. PEP 3152 (and all generator based coroutines) have the limitation that they can't suspend if there's a *Python* function on the stack. Can you see why I know consider this approach categorically worse than one that pursued the Lua approach? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
This means that, whenever a generator yields, that stack needs to be unwound, suspending each affected generator in turn, strung together by references between the generator objects rather than remaining a true frame stack.
The chain of generator frames that results from nested yield-from calls *is* a stack of frames -- it's just linked in the opposite order from the way frames are usually arranged. The "top" of the stack is at the tail, making it slightly less convenient to get to, but that's just an issue of implementation efficiency. -- Greg
Nick Coghlan wrote:
PEP 3152 (and all generator based coroutines) have the limitation that they can't suspend if there's a *Python* function on the stack. Can you see why I know consider this approach categorically worse than one that pursued the Lua approach?
Can you give a concrete example of this problem? Even a toy example will do. The only thing I can think of is something like this: def function(co): value = 1000 for i in (1, 2, 3): value -= co.send(i) return value def coroutine(): value = (yield 1) while True: value += 1 print("suspending...") value += (yield value) print("waking...")
co = coroutine() co.send(None) 1 co.send(3) suspending... 4 function(co) waking... suspending... waking... suspending... waking... suspending... 972 co.send(10) waking... suspending... 24
But as far as I understand it, that seems to show the coroutine suspending even though a function is on the stack. Can you explain what you mean and/or how I have misunderstood? -- Steven
Mark Shannon wrote:
In summary, the cofunction proposal is a work-around for a limitation in the VM. By fixing the VM we can have proper coroutines. Surely, it is better to make the VM support the features we want/need rather than bend those features to fit the VM?
The original version of Stackless worked by "flattening" the interpreter the way you suggest, but it was considered far too big an upheaval to consider incorporating into CPython, especially since a lot of extension code would also have to have been rewritten to make it coroutine-friendly. -- Greg
Nick Coghlan wrote:
The limitation of Lua style coroutines is that they can't be suspended from inside a function implemented in C. Without greenlets/Stackless style assembly code, coroutines in Python would likely have the same limitation.
PEP 3152 (and all generator based coroutines) have the limitation that they can't suspend if there's a *Python* function on the stack. Can you see why I know consider this approach categorically worse than one that pursued the Lua approach?
Ouch, yes, point taken. Fortunately, I think I may have an answer to this... Now that the cocall syntax is gone, the bytecode generated for a cofunction is actually identical to that of an ordinary function. The only difference is a flag in the code object. If the flag were moved into the stack frame instead, it would be possible to run any function in either "normal" or "coroutine" mode, depending on whether it was invoked via __call__ or __cocall__. So there would no longer be two kinds of function, no need for 'codef', and any pure-Python code could be used either way. This wouldn't automatically handle the problem of C code -- existing C functions would run in "normal" mode and therefore wouldn't be able to yield. However, there is at least a clear way for C-implemented objects to participate, by providing a __cocall__ method that returns an iterator. -- Greg
On Sat, Oct 29, 2011 at 5:37 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If the flag were moved into the stack frame instead, it would be possible to run any function in either "normal" or "coroutine" mode, depending on whether it was invoked via __call__ or __cocall__.
So there would no longer be two kinds of function, no need for 'codef', and any pure-Python code could be used either way.
This wouldn't automatically handle the problem of C code -- existing C functions would run in "normal" mode and therefore wouldn't be able to yield. However, there is at least a clear way for C-implemented objects to participate, by providing a __cocall__ method that returns an iterator.
Ah, now we're getting somewhere :) OK, so in this approach, *any* Python function could potentially be a coroutine - it would depend on how it was invoked rather than any inherent property of the function definition. An ordinary call would be unchanged, while a cocall would set a flag on the new stack frame to say that this is a coroutine. Yes, I think that's a good step forward. The behaviour of call() syntax in the eval loop would then depend on whether or not the flag was set on the frame. This would add a tiny amount of overhead to all function calls (to check the new flag), but potentially *does* solve the language bifurcation problem (with some additional machinery). However, I think we still potentially have a problem due to the overloading of a single communications channel (i.e. using 'yield' both to suspend the entire coroutine, but also to return values to the next layer out). To illustrate that, I'll repeat the toy example I posted earlier in the thread: # The intervening function that we want to "just work" as part of a coroutine def print_all(iterable): for item in iterable: print(item) # That means we need "iterable" to be able to: # - return items to 'print_all' to be displayed # - suspend the entire coroutine in order to request more data # Now, consider if our 'iterable' was an instance of the following generator def data_iterable(get_data, sentinel=None): while 1: x = get_data() if x is sentinel: break yield x In coroutine mode, the for loop would implicitly invoke iterable.__next__.__cocall__(). The __cocall__ implementations on generator object methods are going to need a way to tell the difference between requests from the generator body to yield a value (which means the __cocall__ should halt, and the value be returned) and requests to suspend the coroutine emanating from the "get_data()" call (which means the __cocall__ should yield the value provided by the frame). That's where I think 'coyield' could come in - it would tell the generator __cocall__ functionality it should pass the suspension request up the chain instead of returning the value to the caller of next/send/throw. This could also be done via a function and a special object type that the generator __cocall__ implementation recognised. The limitation would then just be that any operation invoked via __call__ (typically a C function with no __cocall__ method) would prevent suspension of the coroutine. The end result would actually look a lot like Lua coroutines, but with an additional calling protocol that allowed C extensions to participate if they wanted to. I think eventually the PEP should move towards a more explanatory model: - "coroutines are a tool for lightweight cooperative multitasking" - "coroutines are cool for a variety of reasons (aka people didn't create Twisted, Stackless, greenlets and gevent just for fun)" - "here's how this PEP proposes this functionality should look (aka Lua's coroutines look pretty nice)" - "rather than saving and switching entire stack frames as a unit, Python's generator model supports coroutines by requiring that every frame on the stack know how to suspend itself for later resumption" - "doing this explicitly can be quite clumsy, so this PEP proposes a new protocol for invoking arbitrary functions in a way that sets them up for cooperative multitasking rather than assuming they will run to completion immediately" - "this approach allows C extensions to play nicely with coroutines (by implementing __cocall__ appropriately), but doesn't require low level assembly code the way Stackless/greenlets do" Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 29 October 2011 11:22, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think eventually the PEP should move towards a more explanatory model: - "coroutines are a tool for lightweight cooperative multitasking" - "coroutines are cool for a variety of reasons (aka people didn't create Twisted, Stackless, greenlets and gevent just for fun)" - "here's how this PEP proposes this functionality should look (aka Lua's coroutines look pretty nice)" - "rather than saving and switching entire stack frames as a unit, Python's generator model supports coroutines by requiring that every frame on the stack know how to suspend itself for later resumption" - "doing this explicitly can be quite clumsy, so this PEP proposes a new protocol for invoking arbitrary functions in a way that sets them up for cooperative multitasking rather than assuming they will run to completion immediately" - "this approach allows C extensions to play nicely with coroutines (by implementing __cocall__ appropriately), but doesn't require low level assembly code the way Stackless/greenlets do"
That looks reasonable. There's one further piece of the puzzle I'd like to see covered, though - with the language support as defined in the PEP, can the functionality be implemented library-style (like Lua) or does it need new syntax? The PEP should discuss this, and if syntax is needed, should propose the appropriate syntax. I think that if the runtime support can be built in a way that allows a Lua-style function/method approach, then that should be the initial design, as it's easier to tweak a functional API than to change syntax. If experience shows that code would benefit from syntax support, add that later. Paul.
On Sat, Oct 29, 2011 at 8:51 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I think that if the runtime support can be built in a way that allows a Lua-style function/method approach, then that should be the initial design, as it's easier to tweak a functional API than to change syntax. If experience shows that code would benefit from syntax support, add that later.
I have one specific reason I think the new yield variant should get a new keyword: it's a new kind of flow control, and Python has a history of trying to keep flow control explicit (cf. the PEP 343 with statement discussions). At the *call* level, it wouldn't need a keyword, since cocall(x) could just be a wrapper around x.__cocall__(). This would be similar to the situation with generators - yielding from a generator has dedicated syntax (in the form of 'yield'), but invoking one just uses the standard call and iterator syntax. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 29 October 2011 15:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sat, Oct 29, 2011 at 8:51 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I think that if the runtime support can be built in a way that allows a Lua-style function/method approach, then that should be the initial design, as it's easier to tweak a functional API than to change syntax. If experience shows that code would benefit from syntax support, add that later.
I have one specific reason I think the new yield variant should get a new keyword: it's a new kind of flow control, and Python has a history of trying to keep flow control explicit (cf. the PEP 343 with statement discussions).
That's a reasonable point, and could be explicitly noted in the PEP. Paul.
Paul Moore wrote:
There's one further piece of the puzzle I'd like to see covered, though - with the language support as defined in the PEP, can the functionality be implemented library-style (like Lua) or does it need new syntax?
While it might not need new syntax, it does hinge fundamentally on changing the way the interpreter loop works, so I don't think a library implementation would be possible. -- Greg
Nick Coghlan wrote:
However, I think we still potentially have a problem due to the overloading of a single communications channel (i.e. using 'yield' both to suspend the entire coroutine, but also to return values to the next layer out).
Bugger, you're right. With sufficient cleverness, I think it could be made to work, but it occurs to me that this is really a special case of a more general problem. For example, an __add__ method implemented in Python wouldn't be able to suspend a coroutine. So we would be over-promising somewhat if we claimed that *any* pure-python code could be suspended with 'coyield'. It would be more like "any pure-python code, as long as no special method call is involved anywhere along the call chain". Fixing that would require making some rather extensive changes. Either every type slot would need to get a cocall counterpart, or their signatures would need an argument to indicate that a cocall was being made, or something like that. -- Greg
On Sun, Oct 30, 2011 at 4:02 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
However, I think we still potentially have a problem due to the overloading of a single communications channel (i.e. using 'yield' both to suspend the entire coroutine, but also to return values to the next layer out).
Bugger, you're right.
With sufficient cleverness, I think it could be made to work, but it occurs to me that this is really a special case of a more general problem. For example, an __add__ method implemented in Python wouldn't be able to suspend a coroutine.
So we would be over-promising somewhat if we claimed that *any* pure-python code could be suspended with 'coyield'. It would be more like "any pure-python code, as long as no special method call is involved anywhere along the call chain".
Fixing that would require making some rather extensive changes. Either every type slot would need to get a cocall counterpart, or their signatures would need an argument to indicate that a cocall was being made, or something like that.
Ouch, I'd missed that completely. Supporting a few assembly hacks in the core isn't looking so bad at this point ;) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Greg Ewing wrote:
Paul Moore wrote:
There's one further piece of the puzzle I'd like to see covered, though - with the language support as defined in the PEP, can the functionality be implemented library-style (like Lua) or does it need new syntax?
While it might not need new syntax, it does hinge fundamentally on changing the way the interpreter loop works, so I don't think a library implementation would be possible.
You are correct in saying that it would require a change to the interpreter loop, but that does mean it needs new syntax. I still think that a Coroutine class is the cleanest interface. It seems to me that the greenlets approach may have to be the way forward. IMHO the yield-from approach is rather clunky. I think that flattening the interpreter is the most elegant solution, but the C calls presents in all operators and many functions is a killer. Cheers, Mark.
Hi, I found again a misunderstanding. On 29.10.11 04:10, Nick Coghlan wrote:
On Sat, Oct 29, 2011 at 8:40 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Greg Ewing wrote:
Mark Shannon wrote:
Stackless provides coroutines. Greenlets are also coroutines (I think).
Lua has them, and is implemented in ANSI C, so it can be done portably. These all have drawbacks. Greenlets are based on non-portable (and, I believe, slightly dangerous) C hackery, and I'm given to understand that Lua coroutines can't be suspended from within a C function.
My proposal has limitations, but it has the advantage of being based on fully portable and well-understood techniques. If Stackless has them, could we use that code? That's what the greenlets module *is* - the coroutine code from Stackless, lifted out and provided as an extension module instead of a forked version of the runtime.
No, the greenlet code is a subset of stackless. Stackless could remove its greenlet part and become an assembler-free implementation. It would just not get over the C extension problem. But that could then be handled by using the greenlet as an optional external module. (I think I said that before. Just wanted it to appear here) -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Picking that up, too... On 29.10.11 09:37, Greg Ewing wrote:
Nick Coghlan wrote:
The limitation of Lua style coroutines is that they can't be suspended from inside a function implemented in C. Without greenlets/Stackless style assembly code, coroutines in Python would likely have the same limitation.
PEP 3152 (and all generator based coroutines) have the limitation that they can't suspend if there's a *Python* function on the stack. Can you see why I know consider this approach categorically worse than one that pursued the Lua approach?
Ouch, yes, point taken. Fortunately, I think I may have an answer to this...
Now that the cocall syntax is gone, the bytecode generated for a cofunction is actually identical to that of an ordinary function. The only difference is a flag in the code object.
If the flag were moved into the stack frame instead, it would be possible to run any function in either "normal" or "coroutine" mode, depending on whether it was invoked via __call__ or __cocall__.
So there would no longer be two kinds of function, no need for 'codef', and any pure-Python code could be used either way.
This wouldn't automatically handle the problem of C code -- existing C functions would run in "normal" mode and therefore wouldn't be able to yield. However, there is at least a clear way for C-implemented objects to participate, by providing a __cocall__ method that returns an iterator.
What about this idea? I think I just wrote exactly the same thing in another thread ;-) Is it still under consideration? (I missed quite a lot when recovering from my strokes ...) -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer wrote:
Picking that up, too...
On 29.10.11 09:37, Greg Ewing wrote:
If the flag were moved into the stack frame instead, it would be possible to run any function in either "normal" or "coroutine" mode, depending on whether it was invoked via __call__ or __cocall__.
What about this idea? I think I just wrote exactly the same thing in another thread ;-)
Yes, it's the same idea. As you can see, I did consider it at one point, but I had second thoughts when I realised that it wouldn't work through type slots, meaning that there would be some areas of what appear to be pure Python code, but are not suspendable for non-obvious reasons. Maybe this is not a fatal problem -- we just tell people that __xxx__ methods are not suspendable. It's something to consider. -- Greg
participants (9)
-
Christian Tismer -
Ethan Furman -
Greg Ewing -
Mark Shannon -
Nick Coghlan -
Paul Moore -
Ron Adam -
Steven D'Aprano -
Yuval Greenfield