On 27/10/11 15:53, Nick Coghlan wrote:

That idea doesn't work when combined with the current idea in the PEP for implicit cocall support inside cofunction definitions. Simple method calls will see "Oh, this has __cocall__" and try to treat it like a cofunction, when the underlying object is actually an ordinary callable.

That's why I've specified that the __cocall__ method can return NotImplemented to signal that the object doesn't support cocalls. When I say "implements __cocall__" I mean "has a __cocall__ method that doesn't return NotImplemented". The prototype implementation makes use of this -- the only difference between a function and a cofunction in that implementation is a flag on the code object. The function object's __cocall__ method returns NotImplemented if that flag is not set.

However, you still potentially have some pretty serious ambiguity problems - what happens inside an *actual* yield from clause? Or a for loop iterator expression?

There is no ambiguity. If f is a cofunction, then yield from f() is equivalent to yield from (yield from f.__cocall__()) and for x in f(): is equivalent to for x in (yield from f.__cocall__()): Both of these make sense provided that the return value of f() is an iterable. -- Greg

Nick Coghlan wrote:

Could you add a pseudocode expansion, similar to the one in PEP 343?

Something like:

_make_call = True _cocall = getattr(obj, "__cocall__", None) if _cocall is not None: _cocall_result = _cocall(*args, **kwds) _make_call = _cocall_result is NotImplemented if _make_call: _result = obj(*args, **kwds) else: _result = _cocall_result

It's not quite as simple as that, because whether a cocall was done determines whether the return value is subjected do a yield-from or used directly as the result. This could be expressed by saying that result = obj(*args, **kwds) expands into _done = False _cocall = getattr(obj, "__cocall__", None) if _cocall is not None: _iter = _cocall(*args, **kwds) if _iter is not NotImplemented: result = yield from _iter _done = True if not _done: result = obj.__call__(*args, **kwds)

Perhaps the PEP needs a "with not codef:" construct to revert to normal calling semantics for a section of code within a coroutine? You could still explicitly yield from such a code block, but it would otherwise be the coroutine equivalent of a critical section.

This is perhaps worth thinking about, but I'm not sure it's worth introducing special syntax for it. If you're working from the mindset of making as few assumptions as possible about suspension points, then your critical sections should be small and confined to the implementations of a few operations on your shared data structures. Now if you have a class such as class Queue: def add_item(self, x): ... def remove_item(): ... then it's already obvious from the fact that these methods are defined with 'def' and not 'codef' that they can't cause a suspension, either directly or indirectly, and will therefore be executed atomically from the coroutine perspective. In other words, wherever you might feel inclined to enclose a set of statements in a 'with not codef' block, you can get the same effect by factoring them out into an ordinary function, the atomicity of which is easy to verify by inspecting the function header. So the motivation for introducing more special syntax here appears weak. -- Greg

On Thu, Oct 27, 2011 at 6:39 PM, Greg Ewing wrote:

Another cofunctions revision:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cd_current/cof...

I've added a formal expansion for cocalls

I think using _iter as the name for the result of the __cocall__ invocation helps make it clear what is going on - it's probably worth doing the same for the costart() definition as well.

and sharpened up the argument about critical sections to hopefully make it less handwavey and more persuasive.

Definitely less handwavey, but I'm not entirely sold on the "more persuasive" just yet. Consider that Java has a synchronized statement, and then synchronized methods are just a shorthand for wrapping the entire body in a synchronized statement. Consider that explicit locking (either manually or via a with statement) is more common with Python code than hiding the locks inside a decorator. The PEP currently proposes that, for coroutines, it's OK to *only* have the function level locking and no tool for statement level locking. To see why this is potentially an issue, consider the case where you have a function that you've decided should really be a cofunction and you want to update your entire code base accordingly. The interpreter can help in this task, since you'll get errors wherever it is called from an ordinary function, but silent changes of behaviour in cofunctions. The latter is mostly a good thing in terms of making coroutine programming easier (and is pretty much the entire point of the PEP), but it is *not* acceptable if manual auditing is the proposed solution to the 'critical section for implicit cooperative concurrency' problem. After all, one of the big things that makes cooperative threading easier than preemptive threading is that you have well defined points where you may be interrupted. The cofunction PEP in its current form gives up (some of) that advantage by making every explicit function call a potential yield point, but without providing any new tools to help manage that loss of control. That means "self.x += y" is safe, but "self.x = y(self.x)" is not (since y may yield control and self.x may be overwritten based on a call that started with a now out of date value for self.x), and there's no way to make explicit (either to the interpreter or to the reader) that 'y' is guaranteed to be an ordinary function. I honestly believe the PEP would be improved if it offered a way to guarantee ordinary call semantics for a suite inside a cofunction so that if someone unexpectedly turns one of the functions called by that code into a cofunction, it will throw an exception rather than silently violating the assumption that that section of code won't yield control to the coroutine scheduler. Obviously, libraries should never make such a change without substantial warning (since it breaks backwards compatibility), but it would still allow a programmer to make an interpreter-enforced declaration that a particular section of code will never implicitly yield control. I think having such a construct will help qualm many of the fears people had about the original version of the implicit invocation proposal - just as try/except blocks help manage exceptions and explicit locks help manage thread preemption, being able to force ordinary call semantics for a suite would allow people to effectively manage implicit coroutine suspension in cases where they felt it mattered. "with not codef:" was the first spelling I thought of to say "switch off coroutine call semantics for this suite". Another alternative would be "with def:" to say "switch ordinary call semantics back on for this suite", while a third would be to add explicit expressions for *both* "cocall f(*args, **kwds)" *and* "defcall f(*args, **kwds)", such that the choice of "def vs codef" merely changed the default behaviour of call expressions and you could explicitly invoke the other semantics whenever you wanted (although an explicit cocall would automatically turn something into a coroutine, just as yield turns one into a generator). I'm sure there are other colours that bike shed could be painted - my main point is that I think this particular bike shed needs to exist. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sat, Oct 29, 2011 at 8:10 AM, Greg Ewing wrote:

Nick Coghlan wrote:

...

I think having such a construct will help qualm many of the fears people had about the original version of the implicit invocation proposal - just as try/except blocks help manage exceptions and explicit locks help manage thread preemption, being able to force ordinary call semantics for a suite would allow people to effectively manage implicit coroutine suspension in cases where they felt it mattered.

Maybe, but I'm still not convinced that simply factoring out the critical section into a 'def' function isn't sufficient to achieve the same ends of auditability and protection from unexpected changes in library semantics.

Also, elsewhere you're arguing that the ideal situation would be for there to be no distinction at all between normal code and coroutine code, and any piece of code would be able to carry out a coroutine suspension. Would you still want a critical section construct in such a world, and if so, how exactly would it work?

It would trigger a runtime error if any of the called functions attempted to suspend the coroutine. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Arnaud Delobelle wrote:

Hi, I've taken the liberty to translate your examples using a small utility that I wrote some time ago and modified to mimic your proposal (imperfectly, of course, since this runs on unpatched python). FWIW, you can see and download the resulting files here (.html files for viewing, .py files for downloading):

http://www.marooned.org.uk/~arno/cofunctions/

Thanks for your efforts, although I'm not sure how much it helps to see them this way -- the whole point is to enable writing such code *without* going through any great contortions. I suppose they serve as an example of what you would be saved from writing. -- Greg

Paul Moore wrote:

PS On the other hand, this is python-ideas, so I guess it's the right place for blue-sky theorising. If that's all this thread is, maybe I should simply ignore it for 18 months or so... :-)

Yes, the intended audience for the PEP is currently the rather small set of people who have been following the yield-from discussions and understand what it's about. I fully expect that yield-from will need quite some time to bed in before any decisions about something further can be made. I'm thinking a long way ahead here. -- Greg

Sven Marnach wrote:

The usual term in programming is "coroutine" rather than "cofunction":

http://en.wikipedia.org/wiki/Coroutine

That article states that Python has coroutines as of 2.5 -- that's incorrect, isn't it? ~Ethan~

On 27 October 2011 20:15, Arnaud Delobelle wrote:

...

That article states that Python has coroutines as of 2.5 -- that's incorrect, isn't it?

Generator functions + trampoline = coroutines

If I understand, the cofunctions of this thread aren't coroutines as such, they are something intended to make writing coroutines easier in some way. My problem is that it's not at all obvious how they help. That's partly because the generator+trampoline idiom, although possible, is not at all common so that there's little in the way of examples, and even less in the way of common understanding, of how the idiom works and what problems there are putting it into practice. Paul.

Nick Coghlan wrote:

On Fri, Oct 28, 2011 at 5:33 AM, Paul Moore wrote:

...

On 27 October 2011 20:15, Arnaud Delobelle wrote:

...

...

That article states that Python has coroutines as of 2.5 -- that's incorrect, isn't it? Generator functions + trampoline = coroutines If I understand, the cofunctions of this thread aren't coroutines as such, they are something intended to make writing coroutines easier in some way. My problem is that it's not at all obvious how they help. That's partly because the generator+trampoline idiom, although possible, is not at all common so that there's little in the way of examples, and even less in the way of common understanding, of how the idiom works and what problems there are putting it into practice.

I highly recommend reading the article Mark Shannon linked earlier in

If you're talking about this: http://www.jucs.org/jucs_10_7/coroutines_in_lua/de_moura_a_l.pdf I have read it, and while all very interesting, I don't see how it answers the big questions about motivating use-cases for cofunctions as described in this PEP. One specific thing I took out of this is that only the main body of a Python generator can yield. That is, if I write this: def gen(): def inner(): yield 1 yield 2 yield 0 inner() # yield 1 and 2 yield 3 it does not behave as I would like. Instead, I have to write this: def gen(): def inner(): yield 1 yield 2 yield 0 # In Python 3.3, the next 2 lines become "yield from inner()" for x in inner(): yield x yield 3 I can see how that would be a difficulty, particularly when you move away from simple generators yielding values to coroutines that accept values, but isn't that solved by the "yield from" syntax?

the thread. I confess I didn't finish the whole thing, but even the first half of it made me realise *why* coroutine programming in Python (sans Stackless or greenlet) is such a pain: *every* frame on the coroutine stack has to be a generator frame in order to support suspending the generator.

I understand that, or at least I *think* I understand that, but I don't understand what that implies in practice when writing Python code. If all you are saying is that you can't suspend an arbitrary function at at arbitrary point, well, true, but that's a good thing, surely? The idea of a function is that it has one entry point, it does its thing, and then it returns. If you want different behaviour, don't use a function. Or do you mean something else? Actual working Python code (or not working, as the case may be) would probably help.

Ideally, we would like to make it possible to write code like this:

def coroutine_friendly_io(*args, **kwds): if in_coroutine(): return coyield AsychronousRequest(async_op, *args, **kwds) else: return sync_op(*args, **kwds)

Why would you want one function to do double duty as both blocking and non-blocking? Particularly when the two branches don't appear to share any code (at least not in the example as given). To me, this seems like "Do What I Mean" magic which would be better written as a pair of functions: def coroutine_friendly_io(*args, **kwds): yield from AsychronousRequest(async_op, *args, **kwds) def blocking_io(*args, **kwargs): return sync_op(*args, **kwds) -- Steven

Nick Coghlan wrote:

If you can't merge the synchronous and asynchronous version of your I/O routines, it means you end up having to write everything in the entire stack twice - once in an "event loop friendly" way ... and once in the normal procedural way.

Okay, I understand what you're concerned about now. Yes, that's a nuisance, and it would be very nice to be able to avoid it. Unfortunately, I don't know how to implement your suggestion in a fully general way without either resorting to dubious C-level hackery as greenlets do, or turning the entire architecture of CPython inside out as the original version of Stackless did. What I'm trying to do is see how close I can get to the ideal, without hackery, by building on already-established mechanisms in CPython and making as few changes as possible. -- Greg

On Mon, Oct 31, 2011 at 7:16 AM, Greg Ewing wrote:

Nick Coghlan wrote:

...

The fact that greenlets exists (and works) demonstrates that it is possible to correctly manage and execute multiple Python stacks within a single OS thread.

I'm not convinced that greenlets work correctly under all possible circumstances. As I understand, they work by copying pieces of C stack in and out when switching tasks. That fails if a pointer to local storage is kept anywhere that can be reached by a different task. I seem to remember there being an issue with Tk doing something like this and causing problems for Stackless.

Yeah, that sentence should have had a "mostly" inside the parentheses :) However, I think it may be more fruitful to pursue an approach that uses greenlets as the foundation, and works to close the loopholes (e.g. by figuring out a way for C code to mark itself as "coroutine safe" and assuming it is not safe by default) rather than trying to figure out how to make a "generators all the way down" approach handle invocation of arbitrary slots. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Mon, Oct 31, 2011 at 8:12 AM, Matt Joiner wrote:

+10 for greenlet style coroutines. It's a very modern feature and will put python in an excellent position. Also nick your points about asynchronous io being the main use case are spot on as far as I'm concerned.

This whole thread has been very helpful to me in understanding not only why I think Greg's coroutine PEP is important, but also why the whole generators-as-coroutines paradigm feels so restrictive. It means we basically have two potential paths forward: 1. Stackless Python (aka greenlets). Pros: known to work for a large number of use cases Cons: need to define a mechanism to declare that the C stack is being used in a "coroutine friendly" fashion 2. Implicit creation of generator-style suspendable frames Pros: shouldn't need assembly level hackery Cons: effectively requires duplication of every C level type slot with a coroutine friendly equivalent that supports suspension Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Matt Joiner wrote:

I've had a lot of success with the greenlet model, it's quite trivial to wrap it up to implicitly spawn an IO loop under the covers. The downside is that all the client code needs to be adjusted to defer blocking calls to the loop, or the entire standard library must be hooked. Again, this doesn't play well with the C stuff: Native C modules, or any third party calls that don't expect to be part of the greenlet model will block entire threads.

That seems to be true of *any* approach, short of using a hacked libc that replaces all the system calls. -- Greg

On Mon, 2011-10-31 at 21:42 +1300, Greg Ewing wrote:

Thinking about how to support cofunctions iterating over generators that are themselves cofunctions and therefore suspendable, I've come to realise that cofunctionness and generatorness really need to be orthogonal concepts. As well as functions and cofunctions, we need generators and cogenerators.

And thinking about how to allow *that* while building on the yield-from mechanism, I found myself inventing what amounts to a complete parallel implementation of the iterator protocol, in which cogenerators create coiterators having a __conext__ method that raises CoStopIteration when finished, etc, etc... :-\

At which point I thought, well, why not forget about using the iterator protocol as a basis altogether, and design a new protocol specifically designed for the purpose?

About then I also remembered a thought I had in passing earlier, when Nick was talking about the fact that, when using yield-from, there is no object available that can hold a stack of suspended generators, so you end up traversing an ever-lengthening chain of generator frames as the call stack gets deeper.

I keep coming back to resumable exceptions as a way to suspend and resume, separate from the yield data path. Looking up previous discussions about it, there have been requests for them, but for the most part, those specific uses were better handled in other ways. There is also the problem that at a low level, they can be non-determinant as to where exactly the exception actually occurred. These negatives have pretty much killed any further discussion. If we try to make all exceptions resumable, (either at the line they occurred or the line after), there are a lot of questions as to just how to make it work in a nice way. Enough so, it isn't worth doing. But I think there is a way to look at them in a positive light. If we put some strict requirements on the idea. 1. Only have a *SINGLE* exception type as being resumable. 2. That exception should *NEVER* occur naturally. 3. Only allow continuing after it's *EXPLICITLY RAISED* by a raised statement. All of the problem issues go away with those requirements in place, and you only have the issue of how to actually write the patch. Earlier discussions indicated, it might not be that hard to do. Just like ValueError, a ResumableException would/should *never* occur naturally, so there is no issue about where it happened. So as long as they are limited to a single exception type, it may be doable in a clean way. Also they could be sub-classed so in effect offer a multi-channel way to easily control co-routines. The problems with existing designs... I've been playing around with co-function pipes where the data is pulled through. Once you figure out how they work they are fairly easy to design as they are just generators that are linked together in a chain. At the source end is of course some sort of data source or generator, and each link operates on items as they a pulled by a next() call on the other end. In a single pipe design, the last item is also the consumer and pulls things through the pipe as it needs it. But if you need to regulate the timing or run more than one pipe at a time, it requires adding an additional controller generator at the end that serves as part of the framework. The scheduler then does a next() call on the end generator, which causes an item to be pulled through the pipe, The consumer next to the end must push the data somewhere in that case. This type of design has a major problem as the speed the pipe works is determined by how long it takes for data to go through it. Thats not good if we are trying to run many pipes at once. Sense they can only suspend at yields, it requires sending scheduling data through the pipe along with (or instead of) the data and sort that back out at some point. Once we do that, our general purpose generators become tied to the framework. The dual iterator protocol is one way around that. A trampoline can handle this because it sits between every co-function, so it can check the data for signal objects or types that can be handled outside of the coroutines. And then push back the data that it dosn't know how to handle. That works, but it still requires additional overhead to check those messages. And trampolines work by yielding generators, so they do require a bit of studying to understand how they work before you use them. How a ResumableException type would help... With a resumable exception type, we create a data path outside of functions and generators that is easy to parse. So inside the coroutines we only need to add a "raise ResumableException" to transfer control to the scheduler. And then in the scheduler it catches the exception, handles any message it may have, and saves it in a stack. And it can then resume each coroutine by possibly doing a "continue ResumableException". These could be easily sub-classed to create a scheduler ... while 1: thread = stack.popleft() try: continue thread except AddThread as e: ... except RemoveThread as e: ... except Block as e: ... except Unblock as e: ... except ResumableException as e: thread = e stack.append(thread) Where each of those is a sub-class of ResumableException. Notice that, the scheduler is completely separate from the data path, it doesn't need to get the data, test it, and refeed it back in like a trampoline does. It also doesn't need any if-else structure, and doesn't need to access any methods at the python level if the "continue" can take the exception directly. So it should be fast. A plain "raise ResumableExeption" could be auto continued if it isn't caught by default. That way you can write a generator that can work both in threads and by it self. (It's not an error.) There would be no change or overloading of "yield" to make these work as threads. "Yield from" should work as it was described just fine and would be complementary to this. It just makes the whole idea of coroutine based lightweight threads a whole lot easier to use and understand. I think this idea would also have uses in other asynchronous designs. But it definitely should be limited to a single exception type with the requirements I stated above. Cheers, Ron

However, with cofunctions there *is* a place where we could create such an object -- it could be done by costart(). We just need to allow it to be passed to the places where it's needed, and with a brand-new protocol we have the chance to do that.

I'll give a sketch of what such a protocol might be like in my next post.

On Tue, 2011-11-01 at 10:06 +1300, Greg Ewing wrote:

Ron Adam wrote:

...

If we put some strict requirements on the idea.

1. Only have a *SINGLE* exception type as being resumable.

2. That exception should *NEVER* occur naturally.

3. Only allow continuing after it's *EXPLICITLY RAISED* by a raised statement.

All of the problem issues go away with those requirements in place, and you only have the issue of how to actually write the patch. Earlier discussions indicated, it might not be that hard to do.

I'm not familiar with these earlier discussions. Did they go as far as sketching a feasible implementation? It's all very well to propose things like this, but the devil is very much in the details.

Yeah, there isn't very much about the details, but I think it is worth looking into as it would pretty much does exactly what is needed. (IMHO) Here is some of the things I was able to find. But as I said, the discussions didn't get very far. I'm hoping that the idea has more merit in the smaller 'restricted' context of coroutines rather than for general exception handling. Here is a very old 1994 thread that is interesting reading. (Long but worth while.) http://groups.google.com/group/comp.lang.python/browse_thread/thread/674a821... There was a reference to allow 'raise' to send out a reference, rather than an exception. That may be an interesting way to do this. Some more that didn't go anywhere... http://bytes.com/topic/python/answers/46053-resume-after-exception http://bytes.com/topic/python/answers/36650-exception-feature-creep-entering... http://mail.python.org/pipermail/python-list/2010-December/1261919.html I can't seem to find where I found the "It wouldn't be too hard to do part.". But if a final restriction of only working with generators at first is added, it may make it easier as they can already be suspended. Here is a python implementation for lisp style restarts. I haven't studied it yet, but it may show a way. (I'm going to look at this in more detail tonight.) http://pypi.python.org/pypi/withrestart/0.2.6 Not sure if these would help, but they may be of interest on a more theoretical level. http://okmij.org/ftp/continuations/generators.html http://lambda-the-ultimate.org/node/1544 Wikipedia has this on it's exception handling page ... """ Restarts separate mechanism from policy Condition handling moreover provides a separation of mechanism from policy. Restarts provide various possible mechanisms for recovering from error, but do not select which mechanism is appropriate in a given situation. That is the province of the condition handler, which (since it is located in higher-level code) has access to a broader view. """ In the case of coroutines, the error's are the suspension points, and the error handler is the scheduler that switches between them. The context is a bit different but I believe the concept is still applicable. Hope some of this is helpful. Cheers, Ron

On Tue, 2011-11-01 at 11:14 +1000, Nick Coghlan wrote:

On Tue, Nov 1, 2011 at 10:22 AM, Ron Adam wrote:

...

On Tue, 2011-11-01 at 10:06 +1300, Greg Ewing wrote:

...

Ron Adam wrote:

...

If we put some strict requirements on the idea.

1. Only have a *SINGLE* exception type as being resumable.

2. That exception should *NEVER* occur naturally.

3. Only allow continuing after it's *EXPLICITLY RAISED* by a raised statement.

All of the problem issues go away with those requirements in place, and you only have the issue of how to actually write the patch. Earlier discussions indicated, it might not be that hard to do.

I'm not familiar with these earlier discussions. Did they go as far as sketching a feasible implementation? It's all very well to propose things like this, but the devil is very much in the details.

Yeah, there isn't very much about the details, but I think it is worth looking into as it would pretty much does exactly what is needed. (IMHO)

It gave me another thought on an existing utility worth exploring in this context: pdb's post-mortem capabilities.

Now, those *don't* implement coroutines (when you do a postmortem, you end up in an emulation of the eval loop, not the eval loop itself). However, that exception instance *does* contain the full frame stack, all the way down to where the exception was thrown. Figuring out what hooks you would need in the core eval loop in order to reinstate an exception's frame stack as the "real" frame stack might be an interesting exercise.

Poking around a bit, it looks like 'raise' does most of the work and the exception is just an envelope for what ever 'raise' puts in it. Is that right? I'd like to be able to make this work. class Suspend: def __init__(self, source): self.source = source self.suspend = True def __next__(self): nonlocal self.suspend if self.suspend: self.suspend = False raise SuspendException self.suspend = True return next(self.source) There are two issues with it... The "self.suspend = False" doesn't seem to work. The __next__ seems to get it's own copies of the attributes at the time the generator is created. And after the SuspendException is raised, a StopIteratoion is issued on the next next() call. The StopIteration is from the whole chain. The only reason the scheduler doesn't stop is it catches the Suspendexception. I want to be able to stick something like this in the generator chained pipe example below. Cheers, Ron *This is broken down into finer steps than you would normally do in order to test how it behaves. """ Basic scheduler test -- co-pipes version """ from co_pipes import * def Person(args): name, count = args p = Producer(lambda:name) # call function each time p = Limit(p, count) # exit after count yields. p = Enumerate(p) # -> (n, data) #p = Suspend(p) # suspend doesn't work. p = Apply(p, "{0[0]}: {0[1]} ".format) p = Apply(p, print) # consumer for _ in p: yield # pull data from here. def main(data): p = Source(data) # take an iterable p = Apply(p, Person) # apply a callable to data p = Collect(p) # collect items in a list p = Scheduler(p) # take a list of generators. next(p) # start the scheduler. if __name__ == "__main__": data = [("John", 2), ("Micheal", 3), ("Terry", 4)] main(data) Prints... 1: John 1: Micheal 1: Terry 2: John 2: Micheal 2: Terry 3: Micheal 3: Terry 4: Terry

On 11/1/2011 12:58 AM, Ron Adam wrote:

Poking around a bit, it looks like 'raise' does most of the work and the exception is just an envelope for what ever 'raise' puts in it. Is that right?

I believe raise just instantiates the indicated exception. I expect that Exception.__new__ or .__init__ captures the traceback info. Subclasses can add more. A SuspendExecution exception should be able to grab as much as is needed for a resume. A CAPI call could be added if needed. I hope you keep looking at this idea. Function calls stop execution and pass control 'down', to be resumed by return. yield stops execution and passes control 'up', to be resumed by next (or .send). Exceptions pass control 'up' (or 'out') without the possibility of resuming. All that is lacking is something to suspend and pass control 'sideways', to a specific target. A special exception makes some sense in that exceptions already get the call stack needed to resume after suspension. -- Terry Jan Reedy

A Coroutine Protocol ==================== Here are some thoughts on the design of a new protocol to support lightweight threads using a mechanism similar to, but distinct from, generators and yield-from. Separating the two protocols will make it much easier to support suspendable generators, something that is not possible using the cofunction mechanism as currently specified in PEP 3152. The protocol to be described is similar in many ways to the generator protocol, and in what follows, analogies will be drawn between the two protocols where it may aid understanding. API --- This section describes the outward appearance of the coroutine mechanism to the programmer. A coroutine is created using the following constructor: :: coroutine(f, *args, **kwds) where ``f`` is an object obeying the "coroutine protocol" to be described below. Syntactic support will be provided for creating such an object using a special form of Python function definition, analogous to a generator. The result is a "coroutine object" having the following methods: ``resume(value = None)`` Resumes execution of the coroutine at the point where it was last suspended. The value, if any, is passed into the coroutine and becomes the return value of the operation that caused the suspension. The coroutine executes until its next suspension point, at which time the ``resume`` call returns with the value passed into the suspension operation. (Note: This is analogous to calling next() or send() on a generator-iterator. Suspension of a coroutine is analogous to a generator executing a ``yield`` operation.) If the coroutine has been freshly created, the passed-in value is ignored and the coroutine executes up to its first suspension point. If the top level of the coroutine finishes execution without encountering any further suspension points, a ``CoReturn`` exception is raised. This exception has a ``value`` attribute containing the return value from the coroutine. (Note: ``CoReturn`` is analogous to the ``StopIteration`` exception raised by an exhausted iterator or generator.) ``throw(exception)`` Causes the given exception to be raised in the coroutine at its current suspension point. ``close()`` Requests that the coroutine shut down and clean itself up. This is achieved by throwing in a ``CoExit`` exception (analogous to ``GeneratorExit``). It is expected that programmers will not write code that deals directly with coroutine objects very often; rather, some kind of driver or scheduler will be used that takes care of making ``resume()`` calls and handling ``CoReturn`` exceptions. Cofunctions ----------- There will be a special form of Python function called a "cofunction", defined using the new keyword ``codef`` in place of ``def``. A cofunction provides a convenient way of creating an object obeying the coroutine protocol. (This is similar to how a generator provides a convenient way of creating an object obeying the iterator protocol). Suspension of a cofunction is achieved using the expression :: ``coyield`` [value] This is analogous to a ``yield`` expression in a generator, and like ``yield``, it can both provide and receive a value. However, unlike ``yield``, it is *not* restricted to communicating with the immediate caller. It communicates directly with the ``resume`` method of the coroutine, however deep the nesting of calls is between the ``resume`` call and the ``coyield``. There are some restrictions, however: * A ``coyield`` is only allowed in the body of a cofunction (a function defined with ``codef``), not in any other context. * A cofunction can only be called from the body of another cofunction, not in any other context. Exceptions are raised if any of these restrictions are violated. As a consequence, there must be an unbroken chain of cofunctions (or other objects obeying the cofunction protocol, see below) making up the call stack from the ``resume`` method down to the suspension point. A cofunction may call an ordinary function, but that function or anything called by it will not be able to suspend the coroutine. Note that the class of "ordinary functions" includes most functions and methods written in C. However, it is possible for an object implemented in C to participate in a coroutine stack by implementing the coroutine protocol below explicitly. Coroutine Protocol ------------------ As well as the coroutine object, the coroutine protocol involves three other kinds of objects, "cocallable objects", "coframe objects" and "coiterator objects". A cocallable object has the following method: ``__cocall__(*args, **kwds)`` Initiates a suspendable computation. Returns a coframe object. (This is analogous to the __iter__ method of an iterable object.) May return NotImplemented to signal that the object does not support the coroutine protocol. This enables wrapper objects such as bound methods to reflect whether or not the wrapped object supports the coroutine protocol. A coframe object has the following methods: ``__resume__(costack, value)`` There are two purposes for which this method is called: to continue execution from a suspension point, and to pass in the return value resulting from a nested call to another cocallable object. In both cases, the ``resume`` method is expected to continue execution until the next suspension point, and return the value produced by it. If the computation finishes before reaching another suspension point, ``CoReturn(retval)`` must be raised, where ``retval`` is the return value of the computation. (This method is analogous to the __send__ method of a generator-iterator. With a value of None, it is analogous to the __next__ method of an iterator.) The currently-executing coroutine object is passed in as the ``costack`` parameter. The ``__resume__`` method can make a nested call to another cocallable object ``sub`` by performing: ``return costack.call(sub, *args, **kwds)`` No further calls to this coframe will be made until ``obj`` finishes. When it does, the ``__resume__`` method of this coframe is called with the return value from ``sub``. It is the responsibility of the coframe object to keep track of whether the previous call to its ``__resume__`` method resulted in a suspension or a nested call, and make use of the ``value`` parameter accordingly. ``__throw__(costack, exception)`` Called to throw an exception into the computation. The coframe may choose to absorb the exception and continue executing, in which case ``__throw__`` should return the value produced by the next exception point or raise ``CoReturn`` as for ``__resume__``. Alternatively it may allow the same or a different exception to propagate out. Implementation of this method is optional. If it is not present, the behaviour is as if a trivial ``__throw__`` method were present that simply re-raises the exception. A coiterator is an iterator that permits iteration to be carried out in a suspendable manner. A coiterator object has the following method: ``__conext__()`` Returns a coframe for computing the next item from the iteration. This is the coroutine equivalent of an iterator's ``__next__`` method, and behaves accordingly: its ``__resume__`` method must return an item by raising ``CoReturn(item)``. To finish the iteration, it raises ``StopIteration`` as usual. To support coiteration, whenever a "next" operation is invoked by a cofunction (whether implicitly by means of a for-loop or explicitly by calling ``next()``) a ``__conext__`` method is first looked for, and if found, the operation is carried out suspendably. Otherwise a normal call is made to the ``__next__`` method. Formal Semantics ---------------- The semantics of the coroutine object are defined by the following Python implementation. :: class coroutine(object): # Public methods def __init__(self, main, *args, **kwds): self._stack = [] self._push(_cocall(main, *args, **kwds)) def resume(self, value = None): return self._run(value, None) def throw(self, exc): return self._run(None, exc) def close(self): try: self.throw(CoExit) except (CoExit, CoReturn): pass def call(self, subroutine, *args, **kwds): meth = getattr(subroutine, '__cocall__', None) if meth is not None: frame = meth(*args, **kwds) if frame is not NotImplemented: self._push(frame) return self._run(None, None) return CoReturn(subroutine(*args, **kwds)) # Private methods def _run(self, value, exc): while True: try: frame = self._top() if exc is None: return frame.__resume__(self, value) else: meth = getattr(frame, '__throw__', None) if meth is not None: return meth(self, exc) else: raise exc except BaseException as exc: if self._pop(): if isinstance(exc, CoReturn): value = exc.value exc = None else: raise def _push(self, frame): self._stack.append(frame) def _pop(self): if len(self._stack) > 0: del self._stack[-1] return True else: return False def _top(self): return self._stack[-1] -- Greg

On Wed, Nov 2, 2011 at 10:05 AM, Terry Reedy wrote:

...

Cofunctions -----------

There will be a special form of Python function called a "cofunction", defined using the new keyword ``codef`` in place of ``def``.

Is this really needed? The presence of 'coyield' signals 'cofunction', just as 'yield' signals 'generator'. Does a cofunction without a suspend point make sense? (And if it did, 'if False: coyield' or 'coyield' after 'return' could serve as a signal.)

Something is needed, since there probably won't *be* an explicit coyield in the top level function (instead, it would call async I/O operations that used coyield internally). However, as per the previous thread, I don't believe this needs to be embedded in the bytecode by the compiler - it could instead be a runtime switch in the eval loop, changing the way function calls and iteration are handled. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On 2011-11-02 10:54, Greg Ewing wrote:

Nick Coghlan wrote:

...

However, as per the previous thread, I don't believe this needs to be embedded in the bytecode by the compiler - it could instead be a runtime switch in the eval loop, changing the way function calls and iteration are handled.

Yes, but I'm no longer sure whether it's such a good idea to have no special syntax at all to mark a cofunction, seeing as cofunctionness won't be able to propagate through C calls, special methods, etc.

By having cofunctions declared in a distinctive way, you can look at the source and see exactly where the boundary is between cofunction and non-cofunction code. Without such markers, when you get an exception because you tried to suspend in a non-coroutine zone, it may not be obvious at which point along the call chain you made a mistake.

If the switch Nick describes is available as a flag on the frame objects, it would be easy to extend the traceback to show *exactly* where you entered the no-coroutine zone that you are now failing to suspend. I don't think the additional syntax is helpful, and it would be quite annoying to need to have two versions of every wrapper function/decorator to make it useable in both contexts. - Jacob

On Wed, Nov 2, 2011 at 9:54 PM, Jacob Holm wrote:

If the switch Nick describes is available as a flag on the frame objects, it would be easy to extend the traceback to show *exactly* where you entered the no-coroutine zone that you are now failing to suspend.  I don't think the additional syntax is helpful, and it would be quite annoying to need to have two versions of every wrapper function/decorator to make it useable in both contexts.

Indeed, this is an error reporting problem, not something to be solved by maintaining the bifurcation of the language into "coroutine-friendly" and "normal". Again, I go back to the (lack of) contrast between code being run in the main process thread and code being run as part of a separate OS level thread. There, the "boundary" between the two is covered in the way the exception is reported:

...

...

f() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in f Exception threading.Thread(target=f).start() Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python3.2/threading.py", line 736, in _bootstrap_inner self.run() File "/usr/lib64/python3.2/threading.py", line 689, in run self._target(*self._args, **self._kwargs) File "<stdin>", line 2, in f Exception

Heck, Thread + input Queue + output Queue may be an easier to understand conceptual model for coroutine programming than generators. While the internals would be completely different (since the whole point of coroutines is to avoid the OS level thread overhead), it may help to give people the correct mental model of what is going on.

...

...

def f(): ... print("Started") ... data = cothread.suspend(1) ... print("Resumed:", data) ... return 42 ... cf, data = cothread.cocall(f) Started data 1 cf.resume() Resumed: None Traceback (most recent call last): File "<stdin>", line 1, in <module> File "cothread.py", line 80, in resume return self._wait_for_output() File "cothread.py", line 66, in _wait_for_output raise CoroutineReturn(data.args[0]) cothread.CoroutineReturn: 42

...

...

cf, data = cothread.cocall(f) Started cf.throw(Exception) Traceback (most recent call last): File "cothread.py", line 34, in run result = self._target(*self._args, **self._kwargs) File "<stdin>", line 3, in f File "cothread.py", line 6, in suspend return current.suspend(*args, **kwds) File "cothread.py", line 98, in suspend raise exc Exception

The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "cothread.py", line 89, in throw return self._wait_for_output() File "cothread.py", line 70, in _wait_for_output raise type(exc)(*exc.args) from exc Exception -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Terry Reedy wrote:

Is this really needed? The presence of 'coyield' signals 'cofunction', just as 'yield' signals 'generator'.

The 'coyield' doesn't have to be directly in that function, it could be in something called by that function, any number of levels deep. However, it's since occurred to me that 'coyield' doesn't have to be a keyword, it could be a built-in cofunction.

...

* A cofunction can only be called from the body of another cofunction, not in any other context.

Except that an initial 'call' from a coroutine.resume is needed to get the first cofunction started ;-).

Yes, but that's not done using the normal call syntax, which is what I'm talking about there (that could perhaps be made clearer). -- Greg

I don't think new keywords should be necessary. A module should be sufficient. Also why CoExit when you have GeneratorExit? Might as well make it CoroutineExit. On Tue, Nov 1, 2011 at 9:24 PM, Greg Ewing wrote:

A Coroutine Protocol ====================

Here are some thoughts on the design of a new protocol to support lightweight threads using a mechanism similar to, but distinct from, generators and yield-from. Separating the two protocols will make it much easier to support suspendable generators, something that is not possible using the cofunction mechanism as currently specified in PEP 3152.

The protocol to be described is similar in many ways to the generator protocol, and in what follows, analogies will be drawn between the two protocols where it may aid understanding.

API ---

This section describes the outward appearance of the coroutine mechanism to the programmer.

A coroutine is created using the following constructor:

::

   coroutine(f, *args, **kwds)

where ``f`` is an object obeying the "coroutine protocol" to be described below. Syntactic support will be provided for creating such an object using a special form of Python function definition, analogous to a generator.

The result is a "coroutine object" having the following methods:

``resume(value = None)``

   Resumes execution of the coroutine at the point where it was last    suspended. The value, if any, is passed into the coroutine and    becomes the return value of the operation that caused the suspension.    The coroutine executes until its next suspension point, at which    time the ``resume`` call returns with the value passed into the    suspension operation.

   (Note: This is analogous to calling next() or send() on a generator-iterator.    Suspension of a coroutine is analogous to a generator executing a    ``yield`` operation.)

   If the coroutine has been freshly created, the passed-in value is    ignored and the coroutine executes up to its first suspension point.

   If the top level of the coroutine finishes execution without    encountering any further suspension points, a ``CoReturn`` exception    is raised. This exception has a ``value`` attribute containing the    return value from the coroutine.

   (Note: ``CoReturn`` is analogous to the ``StopIteration`` exception    raised by an exhausted iterator or generator.)

``throw(exception)``

   Causes the given exception to be raised in the coroutine at its    current suspension point.

``close()``

   Requests that the coroutine shut down and clean itself up. This is    achieved by throwing in a ``CoExit`` exception (analogous to ``GeneratorExit``).

It is expected that programmers will not write code that deals directly with coroutine objects very often; rather, some kind of driver or scheduler will be used that takes care of making ``resume()`` calls and handling ``CoReturn`` exceptions.

Cofunctions -----------

There will be a special form of Python function called a "cofunction", defined using the new keyword ``codef`` in place of ``def``. A cofunction provides a convenient way of creating an object obeying the coroutine protocol. (This is similar to how a generator provides a convenient way of creating an object obeying the iterator protocol).

Suspension of a cofunction is achieved using the expression

::

   ``coyield`` [value]

This is analogous to a ``yield`` expression in a generator, and like ``yield``, it can both provide and receive a value. However, unlike ``yield``, it is *not* restricted to communicating with the immediate caller. It communicates directly with the ``resume`` method of the coroutine, however deep the nesting of calls is between the ``resume`` call and the ``coyield``.

There are some restrictions, however:

* A ``coyield`` is only allowed in the body of a cofunction (a function defined with ``codef``), not in any other context.

* A cofunction can only be called from the body of another cofunction, not in any other context.

Exceptions are raised if any of these restrictions are violated.

As a consequence, there must be an unbroken chain of cofunctions (or other objects obeying the cofunction protocol, see below) making up the call stack from the ``resume`` method down to the suspension point. A cofunction may call an ordinary function, but that function or anything called by it will not be able to suspend the coroutine.

Note that the class of "ordinary functions" includes most functions and methods written in C. However, it is possible for an object implemented in C to participate in a coroutine stack by implementing the coroutine protocol below explicitly.

Coroutine Protocol ------------------

As well as the coroutine object, the coroutine protocol involves three other kinds of objects, "cocallable objects", "coframe objects" and "coiterator objects".

A cocallable object has the following method:

``__cocall__(*args, **kwds)``

   Initiates a suspendable computation. Returns a coframe object.

   (This is analogous to the __iter__ method of an iterable object.)

   May return NotImplemented to signal that the object does not support the    coroutine protocol. This enables wrapper objects such as bound methods to    reflect whether or not the wrapped object supports the coroutine protocol.

A coframe object has the following methods:

``__resume__(costack, value)``

   There are two purposes for which this method is called: to continue    execution from a suspension point, and to pass in the return value resulting    from a nested call to another cocallable object.

   In both cases, the ``resume`` method is expected to continue execution until    the next suspension point, and return the value produced by it. If the    computation finishes before reaching another suspension point,    ``CoReturn(retval)`` must be raised, where ``retval`` is the return value of    the computation.

   (This method is analogous to the __send__ method of a generator-iterator.    With a value of None, it is analogous to the __next__ method of an iterator.)

   The currently-executing coroutine object is passed in as the ``costack``    parameter. The ``__resume__`` method can make a nested call to another cocallable    object ``sub`` by performing:

       ``return costack.call(sub, *args, **kwds)``

   No further calls to this coframe will be made until ``obj`` finishes. When    it does, the ``__resume__`` method of this coframe  is called with the    return value from ``sub``.

   It is the responsibility of the coframe object to keep track of whether the    previous call to its ``__resume__`` method resulted in a suspension or a nested    call, and make use of the ``value`` parameter accordingly.

``__throw__(costack, exception)``

   Called to throw an exception into the computation. The coframe may choose to    absorb the exception and continue executing, in which case ``__throw__`` should    return the value produced by the next exception point or raise ``CoReturn`` as    for ``__resume__``. Alternatively it may allow the same or a different exception    to propagate out.

   Implementation of this method is optional. If it is not present, the behaviour    is as if a trivial ``__throw__`` method were present that simply re-raises the    exception.

A coiterator is an iterator that permits iteration to be carried out in a suspendable manner. A coiterator object has the following method:

``__conext__()``

   Returns a coframe for computing the next item from the iteration. This is the    coroutine equivalent of an iterator's ``__next__`` method, and behaves accordingly:    its ``__resume__`` method must return an item by raising ``CoReturn(item)``. To    finish the iteration, it raises ``StopIteration`` as usual.

To support coiteration, whenever a "next" operation is invoked by a cofunction (whether implicitly by means of a for-loop or explicitly by calling ``next()``) a ``__conext__`` method is first looked for, and if found, the operation is carried out suspendably. Otherwise a normal call is made to the ``__next__`` method.

Formal Semantics ----------------

The semantics of the coroutine object are defined by the following Python implementation.

::

   class coroutine(object):

       #  Public methods

       def __init__(self, main, *args, **kwds):            self._stack = []            self._push(_cocall(main, *args, **kwds))

       def resume(self, value = None):            return self._run(value, None)

       def throw(self, exc):            return self._run(None, exc)

       def close(self):            try:                self.throw(CoExit)            except (CoExit, CoReturn):                pass

       def call(self, subroutine, *args, **kwds):            meth = getattr(subroutine, '__cocall__', None)            if meth is not None:                frame = meth(*args, **kwds)                if frame is not NotImplemented:                    self._push(frame)                    return self._run(None, None)            return CoReturn(subroutine(*args, **kwds))

       #  Private methods

       def _run(self, value, exc):            while True:                try:                    frame = self._top()                    if exc is None:                        return frame.__resume__(self, value)                    else:                        meth = getattr(frame, '__throw__', None)                        if meth is not None:                            return meth(self, exc)                        else:                            raise exc                except BaseException as exc:                    if self._pop():                        if isinstance(exc, CoReturn):                            value = exc.value                            exc = None                    else:                        raise

       def _push(self, frame):            self._stack.append(frame)

       def _pop(self):            if len(self._stack) > 0:                del self._stack[-1]                return True            else:                return False

       def _top(self):            return self._stack[-1]

-- Greg

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Tue, 2011-11-01 at 18:15 +1000, Nick Coghlan wrote:

On Tue, Nov 1, 2011 at 4:27 PM, Terry Reedy wrote:

...

I believe raise just instantiates the indicated exception. I expect that Exception.__new__ or .__init__ captures the traceback info. Subclasses can add more. A SuspendExecution exception should be able to grab as much as is needed for a resume. A CAPI call could be added if needed.

No, the traceback info is added by the eval loop itself. Remember that when you raise an exception *type* (rather than an instance), the exception doesn't get instantiated until it gets caught somewhere - the eval loop maintains the unwinding stack for the traceback as part of the thread state until it is time to attach it to the exception object.

This is all at the tail end of the eval loop in CPython, but be warned it's fairly brain bending stuff that depends on various internal details of the eval loop: http://hg.python.org/cpython/file/default/Python/ceval.c#l2879

Thanks for the link, I've been trying to get my brain bent around it, but, yes it is hard to understand how it all ties together. This morning I had a thought and maybe it may lead somewhere... Would it be possible to rewrite the 'yield' internals so they work in the following way... # a = yield b try: raise SuspendException(b, _self=_self) Except ContinueException as exc: a = exc.args # b = gen.send(a) def send(gen, a=None): try: gen.throw(ContinueException(a)) except SuspendException as exc: (gen, *b) = exc.args return b The two requirements for this to work are... *A SuspendException needs to be able to pass out of the generator without causing it to stop. *A throw needs to be able to work where the SuspendException was raised. The next issue after that is how to allow a subclass of SuspendException to get pass the next() or .send() caller. A subclassed SuspendException would still be caught by 'except SuspendException as exc'. This is needed as a scheduler or other outer framework sits outside the scope the generator is *called in. *Exceptions work in the callers frame rather than the defining scope. That's an important feature as it will allow coroutines much more freedom to be used in different contexts. What this does is give the non_local symantics you mentioned earlier. Cheers, Ron

...

I hope you keep looking at this idea. Function calls stop execution and pass control 'down', to be resumed by return. yield stops execution and passes control 'up', to be resumed by next (or .send). Exceptions pass control 'up' (or 'out') without the possibility of resuming. All that is lacking is something to suspend and pass control 'sideways', to a specific target. A special exception makes some sense in that exceptions already get the call stack needed to resume after suspension.

That's not actually true - due to the need to process exception handling clauses and finally blocks (including the implicit ones inside with statements), the internal state of those frames is potentially no longer valid for resumption (they've moved on beyond the point where the internal function was called).

I'll also note that it isn't necessary to pass control sideways, since there are two different flavours of coroutine design (the PDF article in the other thread describes this well). The Lua version is "asymmetric coroutines", and they only allow you to return to the point that first invoked the coroutine (this model is a fairly close fit with Python's generators and exception handling). The greenlet version is "symmetric" coroutines, and those let you switch directly to any other coroutine.

Both models have their pros and cons, but the main advantage of asymmetric coroutines is that you can just say "suspend this thread" without having to say *where* you want to switch to. Of course, you can implement much the same API with symmetric coroutines as well, so long as you can look up your parent coroutine easily. Ultimately, I expect the symmetric vs asymmetric decision will be driven more by implementation details than by philosophical preferences one way or the other.

I will note that Ron's suggestion to leverage the existing eval loop stack collection provided by the exception handling machinery does heavily favour the asymmetric approach. Having a quick look to refresh my memory of some of the details of CPython's exception handling, I've come to the following tentative conclusions:

- an ordinary exception won't do, since you don't want to trigger except and finally blocks in outer frames (ceval.c#2903) - in CPython, a new "why = WHY_SUSPEND" at the eval loop layer is likely a better approach, since it would allow the frame stack to be collected without triggering exception handling - the stack unwinding would then end when a "SETUP_COCALL" block was encountered on the block stack (just as SETUP_EXCEPT and SETUP_FINALLY can stop the stack unwinding following an exception - with the block stacks within the individual frames preserved, the collected stack should be in a fit state for later restoration - the "fast_yield" code and the generator resumption code should also provide useful insight

There's nothing too magical there - once we disclaim the ability to suspend coroutines while inside a C function (even one that has called back in via the C/Python API), it should boil down to a combination of the existing mechanics for generators and exception handling. So, even though the above description is (highly) CPython specific, it should be feasible for other implementations to come up with something similar (although perhaps not easy: http://lua-users.org/lists/lua-l/2007-07/msg00002.html).

Cheers, Nick.

On Thu, 2011-11-03 at 14:47 +1300, Greg Ewing wrote:

On 03/11/11 05:21, Ron Adam wrote:

...

Would it be possible to rewrite the 'yield' internals so they work in the following way...

# a = yield b try: raise SuspendException(b, _self=_self) Except ContinueException as exc: a = exc.args

I've just been thinking about something like that, while pondering whether there is a middle ground somewhere between the yield-from mechanism and a completely new coroutine protocol.

I was thinking if we could substitute an alternative spelling like that, then it gives us a bit more control on how to interact with other outer frameworks. If the internals can be done that way, then it may open up more options in python generators. So instead of a completely new coroutine proticol, we have better tools for others to create their own frameworks and proticols.

The problem I had with building all of it on yield-from was that there was no way to distinguish a 'yield' being used for the purpose of suspending a coroutine from one being used to return a value to a next() call.

Right. The obvious way is just to add a second of everything. next2() .send2() .throw2() yield2 I don't think that is the best way.

However, if something other than 'yield' is used for coroutine suspension -- such as a 'coyield' keyword or coyield() function -- then I think this problem becomes solvable. In a cogenerator (i.e. a generator running in coroutine mode), 'coyield' would do what 'yield' does in normal mode (simply suspend the frame), and 'yield(value)' would raise StopIteration(value).

Well it sounds reasonable, but how would that actually work? What if the coroutine is paused at coyield, and you need to do a next rather than a conext? And also in the case of it being the othe way round.

(If the latter seems unintuitive, I sympathise. It arises because we're effectively making a cocall to the __next__ method of the generator, and in the yield-from universe, the way a cocall returns a value is by raising StopIteration.)

But now we need a different way for the cogenerator to signal that it has finished iterating! I think what will need to happen is that a cogenerator raises CoStopIteration instead of StopIteration when it falls off the end, and the cocaller of the cogenerator catches that and turns it into a normal StopIteration.

Confused enough yet? I had better finish my cosandwitch and get some more cocoffee before trying to think about this any more...

And that is the whole problem... trying to make this all un_coconfusing to the average python programmer. If it's coconfusing to us, they don't have a chance. ;-) Hmm... I have a craving for some hot co_co now. I've been poking around in genobject.c, frameobject.c, and ceval.c, to try to get a handle on just how it all fits together. One of the odd things is a throw() done before a generator is started rasies an excption at he first line, where it has no chance to act on it. And it also doesn't propagate back out. So the first thing I'm trying (to help me learn the C code better/again) is to see if I can get it to ignore an exception thrown in at that state. And then see if I can make that specific to just g.throw(ContinueException). That way I don't have to pre_start the generators if I'm using throw(ContinueException) as my scheduler interface. It's a start. ;-) Cheers, Ron

On Sun, 2011-11-06 at 13:09 +1300, Greg Ewing wrote:

Ron Adam wrote:

...

On Thu, 2011-11-03 at 14:47 +1300, Greg Ewing wrote:

...

However, if something other than 'yield' is used for coroutine suspension -- such as a 'coyield' keyword or coyield() function -- then I think this problem becomes solvable. In a cogenerator (i.e. a generator running in coroutine mode), 'coyield' would do what 'yield' does in normal mode (simply suspend the frame), and 'yield(value)' would raise StopIteration(value).

Well it sounds reasonable, but how would that actually work? What if the coroutine is paused at coyield, and you need to do a next rather than a conext?

That situation shouldn't occur, because if a generator is suspended at a coyield, it's already in the middle of one next() call, and you shouldn't be trying to start another one until the first one is finished.

Yes, I figured that out just a little before you posted this. A coyield suspends it in between normal yield statements, and you dont want to steel the value from the the next next(), .send(), or .throw() call. Which is why those won't work to continue as well. What is needed is a different path out (and back) to the coroutine that doesn't interfere with the standard yield behavior. Or as you describe it here.

Yes, I'm leaning back towards a completely separate protocol now. We seem to need a number of new protocol features in any case, and allowing the two protocols to overlap is just creating needless confusion, I think.

I think adding a .resume() method to all generators would be good. That keeps it simple. It would just raise an exception in the case of a non-suspended generator. Which may be useful in a loop as well. Looking at ceval.c, I think Nick's suggestion of adding a new WHY_SUSPEND would be good. And along with it, a TARGET(GEN_SUSPEND) block that sets up a SuspendIteration exception, and returns WHY_SUSPEND to the generator object. At that point it could save anything it needs before raising the exception, and when the resume method is called, restore what it needs before calling the frame evel loop. I still haven't quite worked out how to get back to the original next(), .send() or .throw() call. Cheers, Ron

...

And that is the whole problem... trying to make this all un_coconfusing to the average python programmer. If it's coconfusing to us, they don't have a chance. ;-)

On Sun, 2011-11-06 at 22:21 +1000, Nick Coghlan wrote:

On Sun, Nov 6, 2011 at 5:20 PM, Ron Adam wrote:

...

What is needed is a different path out (and back) to the coroutine that doesn't interfere with the standard yield behavior. Or as you describe it here.

Did my post of the thread+queue based implementation of a coroutine API design concept not go through? It demonstrated exactly the need for a separate I/O channel independent of the call/return and next/send/throw/yield channels.

Ok, I found it and the link to the implantation. Thanks for the reminder, I'll check it out tonight when I get back.

...

...

Yes, I'm leaning back towards a completely separate protocol now. We seem to need a number of new protocol features in any case, and allowing the two protocols to overlap is just creating needless confusion, I think.

I think adding a .resume() method to all generators would be good. That keeps it simple. It would just raise an exception in the case of a non-suspended generator. Which may be useful in a loop as well.

Looking at ceval.c, I think Nick's suggestion of adding a new WHY_SUSPEND would be good. And along with it, a TARGET(GEN_SUSPEND) block that sets up a SuspendIteration exception, and returns WHY_SUSPEND to the generator object.

The whole point of a separate WHY_SUSPEND is that you *wouldn't* unwind the stack at all - you'd just leave it in place and return to the point that called into the coroutine in the first place.

Well, that was what I was trying for. In any event, I'm learning a lot more by trying to actually do it, than just speculate about it. I'm at the point of learning just how exceptions interact with the frames. And I'm pretty sure I'll just confirm what you say below, but with a much better understanding of how it all works. Cheers, Ron

Exceptions are too destructive to the stack to be usable for this task (as soon as you hit an except or finally clause, the execution state in the frame gets modified to invoke them, thus meaning you can no longer resume that stack correctly)

Cheers, Nick.

On Tue, Nov 8, 2011 at 3:30 PM, Ron Adam wrote:

Generator API.

  Outside generator   <-->   Inside generator

     next()                     yield      .send()      .throw()

Inverted generator API

  Outside cothread    <-->   Inside cothread

     .resume()                  suspend()                                 throw()

Where resume works like yield, (yield to cothread), and suspend() works like .send().  Throw() raises an exception at the resume() call, like .throw() raises an exception at the yield in a generator.

No, that doesn't make any sense. When the coroutine throws an exception internally it's done - we don't *want* to preserve the stack any more, because something broke and we won't be resuming it. Instead, we let the exception bubble up the stack and if nothing handles it, we pass it back to the thread that called resume(). The reason we need an explicit throw() is that the data request (or whatever it was we suspended to wait for) might fail - in that case, the thread calling resume() needs to be able to indicate this to the cothread by resuming with an exception. The flow control parallels are like this: Inside the generator/cothread: yield -> cothread.suspend() # Wait to be resumed return -> return # Finish normally raise -> raise # Bail out with an error Outside the generator/cothread send() -> resume() # Resume execution normally (optionally providing data) throw() -> throw() # Resume execution with an exception Don't worry about next() in this context, since it's just an alternate spelling for send(None). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Tue, Nov 8, 2011 at 6:24 PM, Ron Adam wrote:

On Tue, 2011-11-08 at 15:46 +1000, Nick Coghlan wrote:

...

No, that doesn't make any sense.

Probably because I didn't explain it well enough.

No, it doesn't make any sense because you're taking a symmetric coroutine concept and attempting to apply it to an asymmetric coroutine API design. *If* this was a symmetric coroutine design (like greenlets), then *yes* it would make sense to offer resume() and throw() as your control flow APIs (there wouldn't be a suspend() at all in a symmetric design, except perhaps as a convenience wrapper around invoking resume() on the coroutine that called resume() or throw() on you). With an asymmetric design, you only have 3 sensible options: terminate naturally, terminate with an exception or suspend execution. Look at the API design again - like the corresponding generator methods, cothread.resume() and cothread.throw() both use "returned normally" to indicate that the coroutine is suspended and still has more to do. If they raise an exception, it means the coroutine is done, either because it failed or because it finished normally (with the latter case being distinguished by a specific exception type, just as it is for generators). You're suggesting it would be reasonable to raise an exception *and* still expect the coroutine to eventually be resumed again. How is the caller of resume() or throw() meant to distinguish that new case from the two terminating cases? Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Ron Adam wrote:

I still haven't quite worked out how to get back to the original next(), .send() or .throw() call.

Keeping the protocols fully separated requires more than just adding methods to the generator. It also requires either its own version of the yield-from chain or some other way of keeping track of the stack of active generators. Once you have that, it becomes clearer how to get back to the right place. -- Greg

Arnaud Delobelle wrote:

Is it worth noting that the implementation in Python (*) that I posted earlier on does just that? The costart generator function, when initiated, keeps track of the stack of suspended generators so that only the current one is accessed at any time.

Yes, I know that this can and has been done before using generators. The important thing is that I'll be using a new protocol that's separate from the generator protocol, making it possible to use both together in a staightforward and efficient way. -- Greg

On Mon, Oct 31, 2011 at 4:54 PM, Greg Ewing wrote:

Or maybe there's a third way:

3. Recognise that, today, some people *are* using generators   to do coroutine-like programming and finding it a useful   technique, despite all of its limitations, and consider   ways to make it easier, even if we can't remove all of   those limitations.

It seems to me that the inability to suspend from within a special method is not likely to be a problem very often in practice.

Doing blocking I/O in special methods (other than __next__) is generally a dubious practice anyway, so you're right that disallowing suspension when such a function is on the stack is unlikely to be a major problem.

Inability to use generators as generators in a coroutine environment may be more serious, since it's quite likely people will want to e.g. use a for-loop to iterate over lines being read from a socket, and suspend while waiting for data to arrive.

I think that particular problem could be solved, given some more thought. The question is, would it be worth the effort?

Yes, I think that one needs to be solved. The issue of providing coroutine-friendly iterative builtins and itertools functionality also need to be addressed. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Steven D'Aprano wrote:

One specific thing I took out of this is that only the main body of a Python generator can yield.

I can see how that would be a difficulty, particularly when you move away from simple generators yielding values to coroutines that accept values, but isn't that solved by the "yield from" syntax?

Only for limited values of "solved" -- to my mind, it's still a rather unnatural way to write coroutine code. Perhaps I should point out that the way I envisage using generators as lightweight threads, the yields will typically be neither sending nor receiving values, but simply serving as suspension points. If a generator is actually producing values, then the phrase "yield from" has meaning, but if it's not -- if you're just using it to calculate a return value or cause a side effect -- it reads like nonsense. Maybe I'm more sensitive to such considerations than other people, but it bothers me. -- Greg

Ron Adam wrote:

Another issue with this is, the routines and the framework become tied together. The routines need to know the proper additional protocol to work with that framework, and they also can't be used with any other framework.

An interesting thing about yield-from is that once you have it, the interface between the generators and the framework reduces to the iterator protocol, and there's really only *one* obvious way to do it. This means it may become feasible to have a single coroutine scheduler in the standard library, with hooks for adding various kinds of event sources, so that different libraries handling asynchronous events can work together, instead of each one wanting to be in charge an run its own event loop. -- Greg

On Fri, Oct 28, 2011 at 8:10 PM, Paul Moore wrote:

On 28 October 2011 00:56, Nick Coghlan wrote:

...

PEP 380 (i.e. "yield from") makes it easier to *create* those stacks of generator frames, but it doesn't make the need for them to go away. Proceeding further down that path (as PEP 3152 does) would essentially partitioning Python programming into two distinct subsets: 'ordinary' programming (where you can freely mix generators and ordinary function frames) and 'generator coroutine' programming (where it *has* to be generators all the way down to get suspension to work).

This comment begs the question, is it the right thing to do to split Python programming into two subsets, as you suggest?

That's actually the status quo - you can right code normally, or you can write it inside out Twisted style.

...

Frankly, now that I understand the problem more clearly, attempting to attack it by making it easier to create stacks consisting entirely of generator frames strikes me as a terrible idea. Far better to find a way to have a dynamic "non-local" yield construct that yields from the *outermost* generator frame in the stack, regardless of whether the current frame is a generator or not.

Hmm, OK. Sort of... I think the effect is correct, but it makes my head hurt having to think about it in terms of the internals of stacks and frames rather than in terms of my code... (I spend a few minutes thinking that the non-local yield should go *to* the generator frame, rather than yield *from* it. And doesn't this make generator loops within generator loops impossible? I don't now if they are needed, but I'd hate to bake in a limitation like that without considering if it's OK.)

Nah, the real beauty of a new mechanism is that it would leave the existing generator channels untouched, so we wouldn't be running *any* risk of breaking it. Generators would still use an ordinary yield and borrow the stack frame of the caller - it's only the new coroutines that would be creating a truly independent frame stack.

...

If you look at the greenlet docs (http://packages.python.org/greenlet/) after reading the article on Lua's coroutines, you'll realise that greenlet implements *symmetric* coroutines - you have to explicitly state which greenlet you are switching to. You can then implement asymmetric coroutines on top of that by always switching to a specific scheduler thread.

Given that the greenlet library exists (and is used by other projects, according to its PyPI page) why all the focus on core support? Seriously, how does the greenlet library fail to provide whatever users need? (Other than "it's not in the core" which could be fixed simply by adding it to the stdlib). I can't see any notes in the greenlet documentation which imply there are limitations/issues that might be a problem.

I believe greenlets supports switching even when there's a C function on the stack, and they use hand written assembly code to do it. So the answer to the question "Why not just use greenlets?" is the following file (and its friends in that directory): https://bitbucket.org/ambroff/greenlet/src/77363116e78d/platform/switch_amd6... That said, *without* that kind of assembly code, we'd probably face the same problem Lua does (i.e. coroutines can't suspend when there's a C function on the frame stack), which could potentially limit the feature's usefulness (e.g. you migrate a function to C or Cython and suddenly your coroutines break).

...

To achieve 'natural' coroutine programming, a Lua style asymmetric coroutine approach looks the most promising to me.

+1. Although I'm not sure whether this needs to be in the core (i.e. with language and/or syntax support), or in the stdlib, or just as a wrapper round the greenlet library.

I'm not sure either - I suspect that without the low level assembly code that greenlets relies on a coroutine library for Python would need support in the core to do the frame stack switching. Years ago, I believe Guido was amenable to the idea of merging changes back from Stackless, maybe he'd be open to the idea of supporting a bit of assembly code in order to get full fledged coroutines in the standard library. It has the usual activation barrier though (i.e. someone seeking the blessing of the greenlets authors, writing a PEP and championing it against all comers). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Thu, Oct 27, 2011 at 10:19 PM, Steven D'Aprano wrote:

The Cofunction Definitions section does not help me understand the concept at all:

(1) You state that it is a "special kind of generator", but don't give any clue as to how it is special. Does it do something other generators can't do? A cofunction is spelled 'codef' instead of 'def', but what difference does that make to the behaviour of the generator you get? How does it behave differently to other generators?

(2) Cofunctions, apparently, "may" contain yield or yield from. Presumably that means that yield is optional, otherwise it would be "must" rather than "may". So what sort of generator do you get without a yield? The PEP doesn't give me any clue.

Yeah, there need to be more references to big motivating use cases/examples of what people are already doing in this space (i.e. Twisted, greenlet, gevent) and how the PEP helps us move towards making asynchronous I/O with an event loop as easy to use as blocking I/O (with or without threads). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Greg Ewing wrote:

Steven D'Aprano wrote:

...

(1) You state that it is a "special kind of generator", but don't give any clue as to how it is special.

You're supposed to read the rest of the specification to find that out.

I did. Twice. It means little to me. The PEP makes too many assumptions about the reader's understanding of the issues involved. If this was a proposal for a new library, I'd say "It doesn't effect me, I can ignore it, I just won't use the library". But you're talking about adding syntax and changing the execution model of Python. This becomes part of the language. I'm not hostile to the idea. But the PEP needs to reference concrete examples that people can run to see that there is a problem to be solved. Otherwise, it just seems that you're proposing adding more complication to the Python, at least one new keyword and one more builtin, for what gain? -- Steven

Steven D'Aprano writes:

I'm not hostile to the idea. But the PEP needs to reference concrete examples that people can run to see that there is a problem to be solved. Otherwise, it just seems that you're proposing adding more complication to the Python, at least one new keyword and one more builtin, for what gain?

For me the first example in the greenlet documentation was very helpful: http://packages.python.org/greenlet/ Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C