On Mon, Aug 23, 2004 at 11:56:04AM -0400, Phillip J. Eby wrote: | >1. The PEP would introduce a new 'builtin' class called 'Cooperate' | > | >2. Generator semantics would be altered so that 'yield X', where X | > is an instance of Cooperate, would automagically propigate to | > the outer-most non-generator. | | Perhaps you mean "inner-most"? Yes. The top-most non-generator on the stack. | It doesn't seem to me to actually help anything. You can already do this | using a simple wrapper object that maintains a stack of active | generators, as I do in 'peak.events'. Could you provide an example? The problem this proposal solves is straight-foward -- it is tedious and slow to have intermediate generators do stuff like: def middle(): """ intermediate generator _only_ sees one and two """ for x in top(): ! if isinstance(x,X): ! yield x print "middle", x yield x This extra step is tedious and also slow; especially if one has lots of yield statements that cooperate. It could be standardized and made a bit snappier if it was built-in behavior. This is an 80/5 proposal. One gets 80% of the happiness, with 5% of the pain. | I was hoping that you had actually come up with a solution for the more | complex problem of suspending *non* generator functions, in a way that | would work with CPython. :( Yes, I know. I'm trying to avoid this much harder problem. Clark

On Mon, Aug 23, 2004 at 01:18:28PM -0400, Phillip J. Eby wrote: | It would be nice if there were some way to "accept" data and exceptions | within a generator that didn't require the 'events.resume' hack, e.g.: | | result = yield aDeferred | | would be really nice, especially if 'result' could cause an exception to | be raised. I was hoping that this was something along the lines of what | you were proposing. Perhaps it would be nice to add an alternative syntax to call a generator when you are expecting exactly one value. def generator(): yield 'one value' def consumer(): value = generator() This, when combined with the previous proposal would give: >>> def top(): >>> yield cooperate >>> yield "one value" >>> >>> def middle(): >>> """ intermediate generator _only_ sees 'one value' """ >>> bing = top() >>> # do something with bing >>> yield bing >>> >>> def lower(): >>> """ this is not a generator, so it sees cooperate """ >>> for x in middle(): >>> print x >>> >>> lower() cooperate one value | Perhaps there should be a "simple coroutines" PEP, that doesn't try to | extend generators into coroutines, but instead treats coroutines as a | first-class animal that just happens to be implemented using some of the | same techniques "under the hood". The problem is maintaining a 'stack' of generators requires that intermediate generators in the stack need a tedious hack (in both peek.event and twisted.flow) for them to work. If we can have python allow messages to be sent from the top-most generator to the inner-most non-generator (via yield cooperate, or some other magic), this difficulty is resolved -- intermediate generators can then be written in a operational style, without icky hacks for managing deferred execution. Full-blown corountines arn't necessary. A small tweak to generators will do. Best, Clark

On Mon, Aug 23, 2004 at 02:59:18PM -0400, Phillip J. Eby wrote: | > def generator(): | > yield 'one value' | value, = generator() Oh, cool. ;) | > Full-blown corountines arn't necessary. A small tweak to | > to generators will do. | | I don't think this is true. Your hypothetical example can't resume | 'top()' after it yields the "co-operate" control value, unless it either | it *has* a stack of generators, or the Python core somehow maintains a | stack of the executing generators. Yes. The idea is for Python to maintain a stack of generators, and when ever the top-most generator yields <magic> this yield skips the intermediate generators on the stack, and goes immediately to the calling function. | So, my point was that since this can already be done in user-level code | with a stack of generators, I don't see the point to adding a facility to | Python to create hidden stacks of generators. The problem is, if you maintain the stack of generators in userland, then each generator on the stack must do special stuff to handle 'cooperate' (in one form or another). You demonstrated an equivalent problem with your peek.event code presented earlier. While, if this is built into the core, intermediate generators can be happily oblivious to the cooperative magic. | Instead, it would be more useful to get rid of the one piece that really | is "magic", by providing a way to pass values or exceptions into a | running generator. I don't like this either. Clark

My comment about "coroutines" was more that Guido previously expressed distaste for adding such a communication mechanism to generators as abusing the concept of a generator just being a way to implement complex iterators.

This makes me think that maybe we want another kind of object, similar to a generator, but designed to be used for side effects rather than to produce values. For want of a better term, let's call it a "cooperator" (so it ends in "ator" like "generator" :-). Actually there will be two objects, a cooperator-function (analogous to a generator-function) and a cooperator-instance (analogous to a generator-iterator). Calling a cooperator-function will return a cooperator-instance, which will obey the cooperator protocol. This protocol consists of a single function run(), which causes the cooperator-instance to perform some amount of work and then stop. If there is more work remaining to be done, run() returns a true value, otherwise it returns a false value. There will be a new statement: suspend for use inside a cooperator-function. The presence of this statement is what distinguishes a cooperator-function from an ordinary function (just as the presence of 'yield' distinguishes a generator from an ordinary function). Execution of 'suspend' will cause the run() method to return with a true value. The next time run() is called, execution will resume after the 'suspend'. This is really all that's needed, but one further piece of syntax may be desirable to make it easier for one cooperator to invoke another: do another_coop() would be equivalent to co = another_coop() while co.run(): suspend (Because it implies a 'suspend', the presence of 'do' would also mark a function as a cooperator-function). Something to note is that there's no requirement for a cooperator- instance to be implemented by a cooperator-function, just as an iterator can be implemented in ways other than a generator. This sidesteps some of the difficulties of mixing Python and C calls, since a Python cooperator-function could invoke a cooperator implemented in C which in turn invokes another Python cooperator- function, etc. The only requirement is that all the objects on the way down obey the cooperator protocol. The drawback is that you always have to know when you're calling another cooperator and use 'do'. But that's no worse than the current situation with generators, where you always have to know you're using an iterator and use 'for... yield'. What think ye all? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

"Phillip J. Eby" :

If you could throw a special kind of exception (or even a regular exception), and call traceback.resume() to pick up execution at the point where it was thrown, whether thrown by a generator or a regular function.

Actually, it probably wouldn't be too hard to make exceptions resumable -- provided all the frames in between are Python. If the exception gets thrown through any C calls, though, resumption becomes impossible. Some other structure is needed to hold the state of a resumable C call. I agree that maintaining a stack of *ators automatically somehow would be desirable, but I haven't figured out yet exactly where and how that stack would be maintained.

What's really needed (IMO) is to add a way to communicate *into* a co-operator, passing it a value to "accept" or a traceback to raise. E.g.:

input = suspend output

There have been many suggestions in the past for 'yield' to be extended to allow values to be passed in as well as out. They all suffer from a problem of asymmetry. However you arrange it, you always end up having to discard the first value passed out or pass a dummy value in the first time, or something like that. I deliberately left communication out of the suspend to avoid those problems. If you want communication, you have to arrange it some other way.

Anyway, as I said, what would be *most* useful for async programming is a way to resume a traceback, because then you wouldn't need for every intervening frame to have special suspension capability.

Certainly, but that way, ultimately, lies Stackless... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

I'm not tied to my proposal, I'm suggesting that if the scope of continuations is limited to only work with iterators, that a solution may emerge. This limitation, in pratice with generator syntax, is not all that bothersome to an application developer. On Wed, Aug 25, 2004 at 12:13:46PM -0400, Phillip J. Eby wrote: | At 02:51 PM 8/25/04 +1200, Greg Ewing wrote: | >> If you could throw a special kind of exception (or even a regular | >> exception), and call traceback.resume() to pick up execution at the | >> point where it was thrown, whether thrown by a generator or a | >> regular function. | > | >Actually, it probably wouldn't be too hard to make exceptions | >resumable -- provided all the frames in between are Python. If the | >exception gets thrown through any C calls, though, resumption becomes | >impossible. Some other structure is needed to hold the state of a | >resumable C call. | | Unfortunately, as was discussed on the previous Stackless thread, nearly | *all* Python bytecodes pass through C on their way to other Python code. However, the C code in the distribution could be scrubbed to make sure that it works with the mechanism. And the C API could be detailed so that this case is managable; else, it is an "uncaught" exception. | I'm going to have to think about this one some more, but it seems to me | that it would be much easier to add bidirectional communication to | generators to create a coroutine type, than to try to implement resumable | exceptions. Sure; but resumable exceptions would only need to operate with generators, and there are some simple rewrite tricks one could do. If an iterator is calling a generator it could be re-written in a form that maintains state and _doesn't_ catch the exception. Already for generators 'finally' blocks are troublesome, so this doesn't change things for the worse. StopIteration has many of these same issues. On Wed, Aug 25, 2004 at 03:32:28PM +1200, Greg Ewing wrote: | > You'd definately want to mix them. For example, a 'cooperator' | > would be reading from a socket, it could do one of three things: | > | > yield the next line from the sender | > raise StopIteration if the socket closed | > raise SuspendIteration if the read() is blocked | | Hmmm. It seems that generation and cooperation are really orthogonal | concepts -- you may want something that is both a generator *and* a | cooperator. My proposal doesn't allow for this. I will have to do some | more thinking... Not only othogonal, but quite complementary. Kind Regards, Clark -- Clark C. Evans Prometheus Research, LLC. http://www.prometheusresearch.com/ o office: +1.203.777.2550 ~/ , mobile: +1.203.444.0557 // (( Prometheus Research: Transforming Data Into Knowledge \\ , \/ - Research Exchange Database /\ - Survey & Assessment Technologies ` \ - Software Tools for Researchers ~ *

On Tue, 24 Aug 2004 17:57:18 +1200, Greg Ewing wrote:

...

My comment about "coroutines" was more that Guido previously expressed distaste for adding such a communication mechanism to generators as abusing the concept of a generator just being a way to implement complex iterators.

This makes me think that maybe we want another kind of object, similar to a generator, but designed to be used for side effects rather than to produce values.

For want of a better term, let's call it a "cooperator" (so it ends in "ator" like "generator" :-). Actually there will be two objects, a cooperator-function (analogous to a generator-function) and a cooperator-instance (analogous to a generator-iterator).

Calling a cooperator-function will return a cooperator-instance, which will obey the cooperator protocol. This protocol consists of a single function run(), which causes the cooperator-instance to perform some amount of work and then stop. If there is more work remaining to be done, run() returns a true value, otherwise it returns a false value.

Tell me if I'm missing something here, but with the exception of your two introductory keywords (a subject I thought was supposed to be sacred, with the intention of keeping the amount of keywords as small as possible,) how much difference is there to generators? Are you saying the run() method, after having been called initially, will return a value indicative of the cooperator's business when called again? It seems like a similar feat could be achieved with custom generators and boolean yield statements. Daniel Bickett

"Clark C. Evans" :

In fact, one could implement it using a SuspendIteration exception, in addition to a StopIteration.

I don't think this can be done quite right by raising an exception in the normal way, because you don't want 'finally' clauses to be triggered.

You'd definately want to mix them. For example, a 'cooperator' would be reading from a socket, it could do one of three things:

yield the next line from the sender raise StopIteration if the socket closed raise SuspendIteration if the read() is blocked

Hmmm. It seems that generation and cooperation are really orthogonal concepts -- you may want something that is both a generator *and* a cooperator. My proposal doesn't allow for this. I will have to do some more thinking... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

A few definitions: iterator: > An iterator is an object returned by an __iter__ with a next() method which produces values. An iterator's state is kept on the heap (not on the stack), thus one could consider it 'resumable'. An iterator ends when it raises an exception, the very special exception, 'StopIteration' is used to signal that the iterator is done producing values, but is not really an error condition. generator: > A syntax for making an iterator, it appears in python source code as a function or method with a 'yield' statement. An implementation is free to optimize this function, but it acts as if it had created an iterator. iterlink: > An iterlink is an association of one iterator, the consumer, to another iterator, the producer. The iterlink contains information about both iterators, and a marker to indicate the state of the consumer when it asked for the next value of the producer. iterstack: > A stack of iterlinks, where the producer of one iterlink is the consumer of another. The top iterator in the iterstack is an iterator which is behaving like a producer, but not a consumer. The bottom iterator in the iterstack is one which is a producer to a non-iterator. That is, the caller of the bottom iterator is a function or method which is not an iterator. Ok. With this in mind, I like your idea of a new keyword, 'suspend'. This keyword could be used by the top iterator in an iterstack. When 'suspend' happens, the entire iterstack, and its runstate, is setaside. An a signal is sent to the caller of the bottom iterator; the signal could be viewed as a 'resumable exception'. For example, it could be pictured as a 'SuspendIteration' exception. More concretely, using a very simple case. class TopIterator: """ def TopIterator(): yield "one" suspend yield "two" """ def _one(self): self.next = self._suspend return "one" def _suspend(self): self.next = self._two raise SuspendIteration() def _two(self): self.next = self._stop return "two" def _stop(self): raise StopIteration def __iter__(): self.next = self._one return self class BottomIterator(): """ def BottomIterator(): producer = iter(TopIterator()) saved_one = producer.next() saved_two = producer.next() yield (saved_one,saved_two) """ def _one(self): self.saved_one = self.producer.next() self.next = self._two return self.next() def _two(self): self.saved_two = self.producer.next() self.next = self._stop return (self.saved_one, self.saved_two) def _stop(self): raise StopIteration def __iter__(self): self.producer = iter(TopIterator()) self.next = self._one return self def caller(): it = iter(BottomIterator()) while True: try: print it.next() except StopIteration: break except SuspendIteration: sleep(10) continue Anyway, the behavior probably equivalent to 'resumable exceptions', however, I don't think it is _different_ item from an iterator, just a new set of behaviors triggered by the 'suspend' keyword. As long as the 'iterator construction' is automated, the 'caller()' function can be delegated to Twisted's reactor, or asynccore or any other cooperative multitasking base. The superclasses for SuspendIteration could be 'BlockForSocket', for example. On Tue, Aug 24, 2004 at 05:57:18PM +1200, Greg Ewing wrote: | This makes me think that maybe we want another kind of object, similar | to a generator, but designed to be used for side effects rather than | to produce values. (a) the generation of 'top-level' iterators using 'suspend' would be quite easy, see above (b) the generation logic for "pass-through" generators (aka BottomIterator in the example above) would have to happen for the mechanism to be truly useful; SuspendIteration would have to be a pass-through exception (c) updating itertools to allow SuspendIteration to 'passthrough' | For want of a better term, let's call it a "cooperator" (so it ends in | "ator" like "generator" :-). Actually there will be two objects, a | cooperator-function (analogous to a generator-function) and a | cooperator-instance (analogous to a generator-iterator). | | Calling a cooperator-function will return a cooperator-instance, which | will obey the cooperator protocol. This protocol consists of a single | function run(), which causes the cooperator-instance to perform some | amount of work and then stop. If there is more work remaining to be | done, run() returns a true value, otherwise it returns a false value. Well, run() need not be specified. It would be implemented by the lower-level 'reactor' code, aka the "caller" in the example | There will be a new statement: | | suspend | | for use inside a cooperator-function. The presence of this statement | is what distinguishes a cooperator-function from an ordinary function | (just as the presence of 'yield' distinguishes a generator from an | ordinary function). Execution of 'suspend' will cause the run() method | to return with a true value. The next time run() is called, execution | will resume after the 'suspend'. Right. Only that I'd allow both 'yield' and 'suspend' in the same function, or it really isn't that useful. | This is really all that's needed, but one further piece of syntax may | be desirable to make it easier for one cooperator to invoke another: | | do another_coop() | | would be equivalent to | | co = another_coop() | while co.run(): | suspend Exactly; this is where the tough part is, the 'intermediate' cooperators. Ideally, all generators (since they keep their state on the stack) would make good 'intermediate' cooperators. | Something to note is that there's no requirement for a cooperator- | instance to be implemented by a cooperator-function, just as an | iterator can be implemented in ways other than a generator. This | sidesteps some of the difficulties of mixing Python and C calls, since | a Python cooperator-function could invoke a cooperator implemented in | C which in turn invokes another Python cooperator- function, etc. The | only requirement is that all the objects on the way down obey the | cooperator protocol. Yes. This is the key, you _only_ attempt to suspend iterators, that is objects that already keep their state in the heap. | The drawback is that you always have to know when you're calling | another cooperator and use 'do'. But that's no worse than the current | situation with generators, where you always have to know you're using | an iterator and use 'for... yield'. Er, well, if the interface was an 'SuspendIteration' then it could be generalized to all iterators. Afterall, while an exception invalidates the stack-frame for the current call to next(), it doesn't prevent the caller from invoking next() once again. If someone calls next() manually, then SuspendIteration would act as a regular exception... and be uncaught. Clark

Thanks for responding Terry. I'm not sure if my exposition was clear. On Mon, Aug 23, 2004 at 08:33:09PM -0400, Terry Reedy wrote: | > 1. The PEP would introduce a new 'builtin' class called 'Cooperate' | > | > 2. Generator semantics would be altered so that 'yield X', where X | > is an instance of Cooperate, would automagically propigate to | > the outer-most non-generator. Suppose I have three 'frames', two generators and one function, the call stack looks like: g2 g1 fn For this to occur, fn has called .next() on g1, which has called .next() on g2. The proposed behavior is that "yield magic.Collaborate" provides the value returned by g1.next() within fn. In particular, it would be equivalent as if every "val = g2.next()" in g1 was rewritten: val = g2.next() if isinstance(val, magic.Cooperate): yield val One could do this in three ways: (a) by a convention above, this is done in twisted.flow, (b) by a bytecode hack, or (c) by modifying eval.c to have equivalent behavior. | 1. This alters and complexifies the current generator concept. Correct; it introduces something similar to a corountine. | 2. This imposes an isinstance(x, magic.Collaborate) function call cost on | every yield in every generator. Ok. So, in addition to causing unexpected results (one would have to check to see what sort of object to debug), it would also cause a performance hit. | 3. 'automagic' it is. The pertinent runtime call stack contains not | generator objects but .next methods (or in CPython, method-wrappers). So I | presume you are proposing that yielded Collaborates magically plunk | themselves into the first frame up the stack not associated such objects. Right, the first frame that is not a generator, aka, not having a yield statement. And yes, it is different semantics. | 4. If the returned to frame does not re-call the passed over suspended | frames, they and anything they hold onto are leaked until gc'ed. No. See the above equivalence. It might take a bit of thinking to make the "C" implementation correct, but the semantics are farily straight-forward and should not create leaks. | Here, the effect you seek could be accomplished by an explicit 'if not | isinstance(x, Collaborate):' before the print statement (or, in general, | the block of code that uses x). But what if top is rewritten as | def top(): return ("one", MyCooperate(), "two") | Do you really want the behavior of middle to change, as with your proposal? Yes. | > >>> def lower(): | > >>> """ this is not a generator, so it sees cooperate """ | > >>> for x in middle(): | > >>> print "lower", x | | Now add 'if isinstance(x, Cooperate): return' and the suspended middle will | be left in limbo. Not very tidy ;-) No, it would be just like a generator had 'if 1: return'; the suspended middle would be cleaned up just as it is today. ... However, you've convinced me on the semantic differences and the performance hit. I guess the proposal would have to present a new keyword, say 'collaborate' and have to have its semantics clearly specified. Thank you _so_ much for your kind review, Clark

Greg Ewing wrote:

...

it would be equivalent as if every "val = g2.next()" in g1 was rewritten:

val = g2.next() if isinstance(val, magic.Cooperate): yield val

Something bothers me about that proposal, and I think it's this: it would mean that generators are no longer simply a particular technique for implementing iterators, but would have special powers that other iterators don't have. Unless recognising the magic Cooperate value and doing the right thing were made part of the iterator protocol, and all iterators were expected to do the right thing with it, which would be a big imposition.

I agree with Greg and the other comments on wanting to keep generators simple. Part of the beauty of generators is the are brain-dead simple to use while still being handy as hell. Playing with them to add the ability to play asynchronous stuff doesn't seem right. I would rather see full-blown coroutines for Python 3000 to deal with this somehow. -Brett