Cofunctions - an enhancement to yield-from
I've been thinking about this idea for a while, and now that I've got yield-from nailed down to my satisfaction, I thought I'd put it out there to see if anyone else thinks it would be a good thing. Cofunctions ----------- A drawback of 'yield from' as a coroutine mechanism is that it must be used every time a function forming part of a coroutine calls another function that can suspend the coroutine, either directly or indirectly. This makes the code read somewhat awkwardly and provides many opportunities for the programmer to make errors. It also introduces considerable coupling, since changing one's mind about whether a function needs to be suspendable requires revisiting all the call sites of that function. This proposal builds on the 'yield from' proposal by introducing a new kind of function that I will call a "cofunction". A cofunction is a special kind of generator, with the following characteristics: - It is defined by using the keyword 'codef' in place of 'def'. - It is always a generator, even if it does not contain any yields. - Whenever a call is made inside a cofunction, it is done using a special COCALL opcode. This first looks for a __cocall__ method on the object being called. If present, it is expected to return an iterable object, which is treated as though 'yield from' had been performed on it. If the object being called does not have a __cocall__ method, or it returns NotImplemented, the call is made in the usual way through the __call__ method. - Cofunctions themselves have a __cocall__ method that does the same thing as __call__. Using these cofunctions, it should be possible to write coroutine code that looks very similar to ordinary code. Cofunctions can call both ordinary functions and other cofunctions using ordinary call syntax. The only special consideration is that 'codef' must be used to define any function that directly or indirectly calls another cofunction. A few subtle details: - Ordinary generators will *not* implement __cocall__. This is so that a cofunction can e.g. contain a for-loop that iterates over a generator without erroneously triggering yield-from behaviour. - Some objects that wrap functions will need to be enhanced with __cocall__ methods that delegate to the underlying function. Obvious ones that spring to mind are bound methods, staticmethods and classmethods. Returning NotImplemented is specified as one of the possible responses of __cocall__ so that a wrapper can report that the wrapped object does not support __cocall__. -- Greg
It does seem like there are two different use cases for generators which the return value semantics of "yield from" starts to make very clear. The original use is building an iterator which produces results in sequence, and the other is the "cofunction" style of executing a sequence of code blocks which might be interleaved with code from the caller and which returns a value like a regular function. I find cofunctions to be a very useful concept. I like the idea of "codef" defining these cofunctions in a way which makes them behave like generators even if they do not contain a yield call. Certainly having to revisit call sites when you shuffle or comment out code is troublesome. What I am suspicious of is automatically making calls to cofunctions from within a cofunction imply 'yield from'. Knowing whether the function call you are making can suspend your code and return to your parent or not is important, since your state might change between when you call some function and when it returns. Instead, I would like to see a different calling mechanism when calling a cofunction, which makes it clear that you know you're calling a cofunction and that this could have implications for your own code. "yield" (and "yield from") serve this purpose somewhat. It seems like calling a normal function using this special calling mechanism could treat them like a cofunction which produced zero iterations and a single return value. Obviously this topic is very close to the monocle framework ( http://github.com/saucelabs/monocle ), where we implement something very much like this as best we can using existing Python functionality. Other projects have tried to use C module stack-swapping coroutines, and I have seen programmers (including the developers of these projects) struggle with the unexpected, thread-like preemption. I believe a word like "yield" makes these cooperation points more obvious. -Greg On Sun, Aug 1, 2010 at 3:09 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
ghazel@gmail.com wrote:
The same situation occurs when dealing with threads, since any function you call could potentially suspend your thread. This doesn't seem to bother people much.
I would like to see a different calling mechanism when calling a cofunction,
The main point of the whole thing is to avoid having to specially mark call sites like this. If you take that away, all that's left is the ability to define a generator without a yield, and I'm not sure it's worth having a whole new kind of function definition just for that. -- Greg
On Sun, Aug 1, 2010 at 5:13 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Exactly, it is the same situation as in threads, when it does not have to be. If you can be preempted at any point, suddenly there is a need for more fine-grained "locking" around state since you can not be sure that some function will not pause your execution and allow something further up the stack to modify it. With a special calling mechanism these cooperative points are very clear, and you can write code to handle potential state changes when the call returns. Instead of locking around state, cooperative points are like unlocking for the duration of the call.
Well, codef is the part I would like to see if anything is added at all. Maybe that's not worth the trouble. -Greg
Hello, Is it an enhancement or really an alternate proposal? From an outsider's view (mine :-)), I think this alternative makes more sense than trying to stretch the generator protocol far beyond what it was designed for at the start. It would also clearly separate the two use cases of generative iteration and coroutines. If so many people think coroutines are important, then it may be time to give them first-class synctatical support in Python. Regards Antoine. On Sun, 01 Aug 2010 22:09:55 +1200 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Antoine Pitrou wrote:
Is it an enhancement or really an alternate proposal?
I would say it's complementary. There are still use cases for 'yield from', when you're dealing with generators that are designed to produce values. Given that both are useful, defining cofunctions in terms of yield-from allows them to share most of the underlying machinery. It also makes it clear how ordinary functions, generators and cofunctions all interact with each other.
Well, that's one way of looking at it. Another is that generators and coroutines are such closely related concepts that it would seem odd not to be able to define one in terms of the other, or both in terms of some unifying construct. -- Greg
On Mon, 02 Aug 2010 12:07:06 +1200 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
It was the tree-walking example, right? It really looked like it solved a non-existing problem.
Well, it sounds like saying classes and functions are closely related concepts because both denote callable objects. I think there is value in clearly distinguished concepts, rather than some intellectually appealing over-generalization (what Joel Spolsky calls Architecture Astronauts). Regards Antoine.
Antoine Pitrou wrote:
Well, it sounds like saying classes and functions are closely related concepts because both denote callable objects.
I think it's more like saying that methods and functions are related, and Python does implement methods in terms of functions. Similarly, cofunctions are a new concept, but they're built out of lower-level pieces -- generators and yield-from -- that are also available separately. (That could be another Python catchphrase, btw: "Batteries available separately." :-) -- Greg
On Mon, Aug 2, 2010 at 10:16 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
For generators, the important benefit of PEP 380 is that it makes sure that nested cleanup happens in the right order. Consider the following currently valid (but subtly broken) generator pair: def receive_message_components(channel): with channel.open() as session: while 1: data = session.wait_for_next() yield data if data == channel.EOM: break def receive_multiple_message_components(server_details, channel_details, limit=None): with Server(server_details) as server: channel = server.channel(channel_details) n = 0 while 1: for component in receive_message_components(channel): yield component if limit is not None: n += 1 if n >= limit: break That code is most likely broken: if an exception (e.g. GeneratorExit) gets thrown into the outer generator, the server connection will be closed while an open session is still using that connection (since the inner generator doesn't get closed until the outer generator's reference to it gets released, by which time the with statement will have killed the server connection). However, if the body of the inner generator were written inline in the outer generator instead then everything would be fine - the session would be closed before the server connection because the exception handling would correctly propagate out from the innermost yield. PEP 380 makes it easy to factor out subgenerators without needing to worry about subtle misbehaviours of exception handling due to the delayed closure of the subgenerators: def receive_multiple_message_components(server_details, channel_details, limit=None): with Server(server_details) as server: channel = server.channel(channel_details) n = 0 while 1: yield from receive_message_components(channel) if limit is not None: n += 1 if n >= limit: break Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 8/1/2010 8:16 PM, Antoine Pitrou wrote:
I disagree with this opinion. I have run into this pattern before and while I didn't need the whole generator protocol honored, I would've benefited just from the fact that the proposed "yield from" flattened the iteration. A while back there was a proposal about adding an IP address manipulation library to stdlib that I took issue with. And so I started to write my own that met the requirements, and when it came time to write __iter__ for the IPNetwork class, it was quite natural to use recursive generators: def __iter__(self): # can I get a "yield from" please?! for v in self._iter_more(self._network_address.number, self._host_mask_bits): yield v def _iter_more(self, number_so_far, zerobits): if zerobits: bit = zerobits[-1] zerobits = zerobits[:-1] # yield all 'x...x0y...y' addresses for v in self._iter_more(number_so_far, zerobits): yield v # yield all 'x...x1y...y' addresses for v in self._iter_more(number_so_far | (1 << bit), zerobits): yield v else: # construct a proper IPAddress instance for the number yield self._address_class(number_so_far) Obviously, it can be flattened by hand, but I doubt it would be as obvious to read later. I have run into this pattern in other cases where I was writing my own (specialized) containers, and would expect others to have as well, unless they were uncomfortable with generators and/or wrote around the problem. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu
On Mon, 02 Aug 2010 12:21:57 -0400 Scott Dial <scott+python-ideas@scottdial.com> wrote:
How would you have benefitted? Is it a problem if the iteration isn't "flattened"? If it's because of the recursion limit, then it's a general problem and I don't think a generator-specific solution is a good idea. If it's an aesthetical preference then I don't think new syntax is warranted for that. Regards Antoine.
On Mon, Aug 2, 2010 at 9:37 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I think the main reason for wanting the stack of yields flattened is the cost of bumping each next() call all the way up and down the stack. Without an optimized yield-from, yield from G is equivalent to "for X in G: yield X" and that means if you have this nested 3 times on the stack, each next() call incurs the overhead of three for-loop iterations. It would be especially bad if you have a fairly deeply nested stack and then the innermost generator yields a large number of values. It remains to be seen at which point this becomes prohibitive and when the overhead of wrapping every generator in a From instance (and passing every next() call through a method of that instance) is actually faster, given that a for-loop iteration is just a few bytecode instructions. -- --Guido van Rossum (python.org/~guido)
On 03/08/10 04:45, Guido van Rossum wrote:
I don't know about the trampoline-style implementations that have been posted, but I did some timings with my yield-from implementation, and it seems that delegating a next() call via yield-from has only about 8% of the overhead of doing the same with a for-loop. I also tried an experiment where I traversed a binary tree using recursive generators and yield-from. For a tree depth of 20, the whole thing was between 2 and 3 times faster than using for-loops. -- Greg
On Sun, Aug 1, 2010 at 3:09 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The only special consideration is that 'codef' must be used to define any function that directly or indirectly calls another cofunction.
It seems to me that this is a big requirement. If a module that I'm using uses cofunctions, then every use of that module must be a cofunction all the way up to my main function. Worse, I wouldn't be able to change the implementation of a module to use cofunctions because it will break the current users. Why do you think this requirement is necessary? --- Bruce http://www.vroospeak.com http://google-gruyere.appspot.com
Bruce Leban wrote:
Well, at least all the way up to the coroutine scheduler or driver or whatever you want to call it. That statement is actually a bit abbreviated. The caller doesn't strictly have to be a cofunction, but if it's not, it has to be aware that the called cofunction is a generator and deal with it as such, for example by using 'yield from' (which in turn makes the caller a generator) or by looping over it (in which case the caller can be an ordinary function). However you arrange things, there has to be an unbroken chain of generators from the main loop driving the coroutines down to the functions containing the yields. The only question is whether to mark them as such at the function level (using 'codef') or at the call site level (using 'yield from').
Worse, I wouldn't be able to change the implementation of a module to use cofunctions because it will break the current users.
There would be nothing to stop a module from using cofunctions internally, as long as it runs its own driver loop and presents its outside interface in the form of ordinary functions that run to completion. But if it exposes any cofunctions directly to its users, they will need to be aware of the fact, just as they need to be aware of an exposed function that is a generator. -- Greg
After mulling it over and trying to understand the thread I still cannot get excited about this proposal. The only concrete objection I have is that it might be hard to implement in Jython or IronPython -- IIRC we were careful to define yield in such a way that it was easy to generate JVM bytecode for them, and IIRC the details of making it easy had to do with the compiler breaking the generator function into different entry points for each resumption point (i.e. after each yield). In a codef you couldn't do this, since you don't know until run time which calls are codef calls. OTOH I do appreciate the desire to reduce the number of places where one has to sprinkle 'yield' over one's code, and I've had a number of situations recently where I had something that logically needed to be a coroutine (to match some API) but just happened not to need any yields, and inevitably my coding went something like (1) forget to put a yield in, (2) frantically debug, (3) slap forehead, (4) add "if 0: yield" to the function, (5) continue with another instance of this, (6) lose sleep over the best place to spell the dummy yield and where to put it. At the same time I don't want to have to mark all my coroutines with a decorator, like Monocle requires (though maybe I should). Finally, regardless of what happens to codef, I am still enthusiastically supporting PEP 380 as it stands, and am excited to see it ported to Python 3.1 (though I hope that once we've done the Mercurial switch, someone will create a branch for it to be merged into 3.3). --Guido On Sun, Aug 1, 2010 at 3:09 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
-- --Guido van Rossum (python.org/~guido)
On 03/08/10 04:39, Guido van Rossum wrote:
The only concrete objection I have is that it might be hard to implement in Jython or IronPython
As long as it's possible to implement 'yield from', it should be possible to implement codef as well. If nothing else, every call could expand into code that checks for the presence of __cocall__ and then performs either a normal call or a yield-from. Another approach would be to compile all calls as yield from cowrap(func, args...) where cowrap is defined something like def cowrap(func, *args, **kwds): if hasattr(func, '__cocall__'): return yield from func.__cocall__(*args, **kwds) else: return func(*args, **kwds) -- Greg
On 03/08/10 04:39, Guido van Rossum wrote:
Would you be interested in a system which requires marking calls to coroutines, but tells you immediately when you have forgotten to mark such a call? It might work something like this: 1. In a cofunction, a call to another cofunction must be marked with 'cocall', e,g. z = cocall f(x, y) 2. Cofunctions *cannot* be called normally -- they do not have a __call__ method, only a __cocall__ method. So if you try to call a cofunction without using cocall, you get an exception. If you try to call an ordinary function using cocall, you get an exception. If you try to use cocall but forget to declare the function with codef, you get an exception (because cocall would only be allowed inside a cofunction). To start things off, a builtin function could be provided such as def costart(f, *args, **kwds): return f.__cocall__(*args, **kwds) which would return an object that a coroutine driver could treat as a generator. I think this scheme would go a long way towards satisfying Antoine's desire to conceptually separate generators and coroutines. It would also enable an implementation to implement coroutines using a different mechanism from generators if it wanted to. -- Greg This email may be confidential and subject to legal privilege, it may not reflect the views of the University of Canterbury, and it is not guaranteed to be virus free. If you are not an intended recipient, please notify the sender immediately and erase all copies of the message and any attachments. Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more information.
On 02Aug2010 23:16, ghazel@gmail.com <ghazel@gmail.com> wrote: | On Mon, Aug 2, 2010 at 11:17 PM, Gregory Ewing | <greg.ewing@canterbury.ac.nz> wrote: | > Would you be interested in a system which requires marking | > calls to coroutines, but tells you immediately when you | > have forgotten to mark such a call? | > | > It might work something like this: | > | > 1. In a cofunction, a call to another cofunction must | > be marked with 'cocall', e,g. | > | > z = cocall f(x, y) | > | > 2. Cofunctions *cannot* be called normally -- they do | > not have a __call__ method, only a __cocall__ method. | > | > So if you try to call a cofunction without using cocall, | > you get an exception. If you try to call an ordinary function | > using cocall, you get an exception. If you try to use cocall | > but forget to declare the function with codef, you get an | > exception (because cocall would only be allowed inside a | > cofunction). | | I like this idea. Having just caught up with this thread, my first thought on reading the opening post was that it could do with (2), above. So a big +1 from me. It avoids misuse by failing early, and makes the failure reason obvious. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ ... you could spend *all day* customizing the title bar. Believe me. I speak from experience. - Matt Welsh
Guido van Rossum wrote:
That would cover part of it, the part about not being able to make a normal call to a cofunction. But it wouldn't enforce only being able to use cocall inside a cofunction, or remove the need for a dummy yield in a cofunction that doesn't otherwise have any. Also, if cocall is just a function, you would still have to use 'yield from' on the result, so all your coroutine calls would end up looking like yield from cocall(f, args) which makes them even more verbose and thumbstickingoutish. -- Greg
On Mon, Aug 2, 2010 at 12:39 PM, Guido van Rossum <guido@python.org> wrote:
We already use decorators to change interfaces so I am +1 on reusing what people already know. contextlib.contextmanager is shorthand for "the following function is shorthand for a contextmanager" from contextlib import contextmanager @contextmanager def myfunc(...): ... So the generator equivalent would be from somelib import generator @generator def myfunc(...): ... Where generator is as simple as def generator(func): def inner(*args, **opts): if False: yield return func(*args, **opts) return inner But the bulk of Greg's proposal is to transform the called function in one of two ways In the original: to make this return func(*args, **opts) equivalent to this yield from func.__cocall__(*args, **opts) # func must be defined with 'codef' or support __cocall__ Or in his second suggested form to make this cocall func(*args, **opts) equivalent to this yield from func.__cocall__(*args, **opts) # func must support __cocall__ I'm not sure if the "codef" keyword is included in the second form. I'm -1 on the first proposal because it buries that the calling function is a generator. "yield from" (which it would be a synonym or replacement for) lets you know the called function is a generator without having to read the body of the called function. I'm -1 on the 2nd form (explicit "cocall") because it is a synonym for "yield from" and "yield from" fits my brain better because reads as "this is yield-like but slightly different." -Jack
Jack Diederich wrote:
Yes, it is. An important part of it is that 'cocall' would only be allowed inside a function defined with 'codef'. Together with the other restrictions, this makes it impossible to mix coroutine and non-coroutine code in invalid ways. I don't think it's possible to get that using yield-from and decorators (at least not without a lot of inefficient hackery). -- Greg
It does seem like there are two different use cases for generators which the return value semantics of "yield from" starts to make very clear. The original use is building an iterator which produces results in sequence, and the other is the "cofunction" style of executing a sequence of code blocks which might be interleaved with code from the caller and which returns a value like a regular function. I find cofunctions to be a very useful concept. I like the idea of "codef" defining these cofunctions in a way which makes them behave like generators even if they do not contain a yield call. Certainly having to revisit call sites when you shuffle or comment out code is troublesome. What I am suspicious of is automatically making calls to cofunctions from within a cofunction imply 'yield from'. Knowing whether the function call you are making can suspend your code and return to your parent or not is important, since your state might change between when you call some function and when it returns. Instead, I would like to see a different calling mechanism when calling a cofunction, which makes it clear that you know you're calling a cofunction and that this could have implications for your own code. "yield" (and "yield from") serve this purpose somewhat. It seems like calling a normal function using this special calling mechanism could treat them like a cofunction which produced zero iterations and a single return value. Obviously this topic is very close to the monocle framework ( http://github.com/saucelabs/monocle ), where we implement something very much like this as best we can using existing Python functionality. Other projects have tried to use C module stack-swapping coroutines, and I have seen programmers (including the developers of these projects) struggle with the unexpected, thread-like preemption. I believe a word like "yield" makes these cooperation points more obvious. -Greg On Sun, Aug 1, 2010 at 3:09 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
ghazel@gmail.com wrote:
The same situation occurs when dealing with threads, since any function you call could potentially suspend your thread. This doesn't seem to bother people much.
I would like to see a different calling mechanism when calling a cofunction,
The main point of the whole thing is to avoid having to specially mark call sites like this. If you take that away, all that's left is the ability to define a generator without a yield, and I'm not sure it's worth having a whole new kind of function definition just for that. -- Greg
On Sun, Aug 1, 2010 at 5:13 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Exactly, it is the same situation as in threads, when it does not have to be. If you can be preempted at any point, suddenly there is a need for more fine-grained "locking" around state since you can not be sure that some function will not pause your execution and allow something further up the stack to modify it. With a special calling mechanism these cooperative points are very clear, and you can write code to handle potential state changes when the call returns. Instead of locking around state, cooperative points are like unlocking for the duration of the call.
Well, codef is the part I would like to see if anything is added at all. Maybe that's not worth the trouble. -Greg
Hello, Is it an enhancement or really an alternate proposal? From an outsider's view (mine :-)), I think this alternative makes more sense than trying to stretch the generator protocol far beyond what it was designed for at the start. It would also clearly separate the two use cases of generative iteration and coroutines. If so many people think coroutines are important, then it may be time to give them first-class synctatical support in Python. Regards Antoine. On Sun, 01 Aug 2010 22:09:55 +1200 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Antoine Pitrou wrote:
Is it an enhancement or really an alternate proposal?
I would say it's complementary. There are still use cases for 'yield from', when you're dealing with generators that are designed to produce values. Given that both are useful, defining cofunctions in terms of yield-from allows them to share most of the underlying machinery. It also makes it clear how ordinary functions, generators and cofunctions all interact with each other.
Well, that's one way of looking at it. Another is that generators and coroutines are such closely related concepts that it would seem odd not to be able to define one in terms of the other, or both in terms of some unifying construct. -- Greg
On Mon, 02 Aug 2010 12:07:06 +1200 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
It was the tree-walking example, right? It really looked like it solved a non-existing problem.
Well, it sounds like saying classes and functions are closely related concepts because both denote callable objects. I think there is value in clearly distinguished concepts, rather than some intellectually appealing over-generalization (what Joel Spolsky calls Architecture Astronauts). Regards Antoine.
Antoine Pitrou wrote:
Well, it sounds like saying classes and functions are closely related concepts because both denote callable objects.
I think it's more like saying that methods and functions are related, and Python does implement methods in terms of functions. Similarly, cofunctions are a new concept, but they're built out of lower-level pieces -- generators and yield-from -- that are also available separately. (That could be another Python catchphrase, btw: "Batteries available separately." :-) -- Greg
On Mon, Aug 2, 2010 at 10:16 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
For generators, the important benefit of PEP 380 is that it makes sure that nested cleanup happens in the right order. Consider the following currently valid (but subtly broken) generator pair: def receive_message_components(channel): with channel.open() as session: while 1: data = session.wait_for_next() yield data if data == channel.EOM: break def receive_multiple_message_components(server_details, channel_details, limit=None): with Server(server_details) as server: channel = server.channel(channel_details) n = 0 while 1: for component in receive_message_components(channel): yield component if limit is not None: n += 1 if n >= limit: break That code is most likely broken: if an exception (e.g. GeneratorExit) gets thrown into the outer generator, the server connection will be closed while an open session is still using that connection (since the inner generator doesn't get closed until the outer generator's reference to it gets released, by which time the with statement will have killed the server connection). However, if the body of the inner generator were written inline in the outer generator instead then everything would be fine - the session would be closed before the server connection because the exception handling would correctly propagate out from the innermost yield. PEP 380 makes it easy to factor out subgenerators without needing to worry about subtle misbehaviours of exception handling due to the delayed closure of the subgenerators: def receive_multiple_message_components(server_details, channel_details, limit=None): with Server(server_details) as server: channel = server.channel(channel_details) n = 0 while 1: yield from receive_message_components(channel) if limit is not None: n += 1 if n >= limit: break Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 8/1/2010 8:16 PM, Antoine Pitrou wrote:
I disagree with this opinion. I have run into this pattern before and while I didn't need the whole generator protocol honored, I would've benefited just from the fact that the proposed "yield from" flattened the iteration. A while back there was a proposal about adding an IP address manipulation library to stdlib that I took issue with. And so I started to write my own that met the requirements, and when it came time to write __iter__ for the IPNetwork class, it was quite natural to use recursive generators: def __iter__(self): # can I get a "yield from" please?! for v in self._iter_more(self._network_address.number, self._host_mask_bits): yield v def _iter_more(self, number_so_far, zerobits): if zerobits: bit = zerobits[-1] zerobits = zerobits[:-1] # yield all 'x...x0y...y' addresses for v in self._iter_more(number_so_far, zerobits): yield v # yield all 'x...x1y...y' addresses for v in self._iter_more(number_so_far | (1 << bit), zerobits): yield v else: # construct a proper IPAddress instance for the number yield self._address_class(number_so_far) Obviously, it can be flattened by hand, but I doubt it would be as obvious to read later. I have run into this pattern in other cases where I was writing my own (specialized) containers, and would expect others to have as well, unless they were uncomfortable with generators and/or wrote around the problem. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu
On Mon, 02 Aug 2010 12:21:57 -0400 Scott Dial <scott+python-ideas@scottdial.com> wrote:
How would you have benefitted? Is it a problem if the iteration isn't "flattened"? If it's because of the recursion limit, then it's a general problem and I don't think a generator-specific solution is a good idea. If it's an aesthetical preference then I don't think new syntax is warranted for that. Regards Antoine.
On Mon, Aug 2, 2010 at 9:37 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I think the main reason for wanting the stack of yields flattened is the cost of bumping each next() call all the way up and down the stack. Without an optimized yield-from, yield from G is equivalent to "for X in G: yield X" and that means if you have this nested 3 times on the stack, each next() call incurs the overhead of three for-loop iterations. It would be especially bad if you have a fairly deeply nested stack and then the innermost generator yields a large number of values. It remains to be seen at which point this becomes prohibitive and when the overhead of wrapping every generator in a From instance (and passing every next() call through a method of that instance) is actually faster, given that a for-loop iteration is just a few bytecode instructions. -- --Guido van Rossum (python.org/~guido)
On 03/08/10 04:45, Guido van Rossum wrote:
I don't know about the trampoline-style implementations that have been posted, but I did some timings with my yield-from implementation, and it seems that delegating a next() call via yield-from has only about 8% of the overhead of doing the same with a for-loop. I also tried an experiment where I traversed a binary tree using recursive generators and yield-from. For a tree depth of 20, the whole thing was between 2 and 3 times faster than using for-loops. -- Greg
On Sun, Aug 1, 2010 at 3:09 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The only special consideration is that 'codef' must be used to define any function that directly or indirectly calls another cofunction.
It seems to me that this is a big requirement. If a module that I'm using uses cofunctions, then every use of that module must be a cofunction all the way up to my main function. Worse, I wouldn't be able to change the implementation of a module to use cofunctions because it will break the current users. Why do you think this requirement is necessary? --- Bruce http://www.vroospeak.com http://google-gruyere.appspot.com
Bruce Leban wrote:
Well, at least all the way up to the coroutine scheduler or driver or whatever you want to call it. That statement is actually a bit abbreviated. The caller doesn't strictly have to be a cofunction, but if it's not, it has to be aware that the called cofunction is a generator and deal with it as such, for example by using 'yield from' (which in turn makes the caller a generator) or by looping over it (in which case the caller can be an ordinary function). However you arrange things, there has to be an unbroken chain of generators from the main loop driving the coroutines down to the functions containing the yields. The only question is whether to mark them as such at the function level (using 'codef') or at the call site level (using 'yield from').
Worse, I wouldn't be able to change the implementation of a module to use cofunctions because it will break the current users.
There would be nothing to stop a module from using cofunctions internally, as long as it runs its own driver loop and presents its outside interface in the form of ordinary functions that run to completion. But if it exposes any cofunctions directly to its users, they will need to be aware of the fact, just as they need to be aware of an exposed function that is a generator. -- Greg
After mulling it over and trying to understand the thread I still cannot get excited about this proposal. The only concrete objection I have is that it might be hard to implement in Jython or IronPython -- IIRC we were careful to define yield in such a way that it was easy to generate JVM bytecode for them, and IIRC the details of making it easy had to do with the compiler breaking the generator function into different entry points for each resumption point (i.e. after each yield). In a codef you couldn't do this, since you don't know until run time which calls are codef calls. OTOH I do appreciate the desire to reduce the number of places where one has to sprinkle 'yield' over one's code, and I've had a number of situations recently where I had something that logically needed to be a coroutine (to match some API) but just happened not to need any yields, and inevitably my coding went something like (1) forget to put a yield in, (2) frantically debug, (3) slap forehead, (4) add "if 0: yield" to the function, (5) continue with another instance of this, (6) lose sleep over the best place to spell the dummy yield and where to put it. At the same time I don't want to have to mark all my coroutines with a decorator, like Monocle requires (though maybe I should). Finally, regardless of what happens to codef, I am still enthusiastically supporting PEP 380 as it stands, and am excited to see it ported to Python 3.1 (though I hope that once we've done the Mercurial switch, someone will create a branch for it to be merged into 3.3). --Guido On Sun, Aug 1, 2010 at 3:09 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
-- --Guido van Rossum (python.org/~guido)
On 03/08/10 04:39, Guido van Rossum wrote:
The only concrete objection I have is that it might be hard to implement in Jython or IronPython
As long as it's possible to implement 'yield from', it should be possible to implement codef as well. If nothing else, every call could expand into code that checks for the presence of __cocall__ and then performs either a normal call or a yield-from. Another approach would be to compile all calls as yield from cowrap(func, args...) where cowrap is defined something like def cowrap(func, *args, **kwds): if hasattr(func, '__cocall__'): return yield from func.__cocall__(*args, **kwds) else: return func(*args, **kwds) -- Greg
On 03/08/10 04:39, Guido van Rossum wrote:
Would you be interested in a system which requires marking calls to coroutines, but tells you immediately when you have forgotten to mark such a call? It might work something like this: 1. In a cofunction, a call to another cofunction must be marked with 'cocall', e,g. z = cocall f(x, y) 2. Cofunctions *cannot* be called normally -- they do not have a __call__ method, only a __cocall__ method. So if you try to call a cofunction without using cocall, you get an exception. If you try to call an ordinary function using cocall, you get an exception. If you try to use cocall but forget to declare the function with codef, you get an exception (because cocall would only be allowed inside a cofunction). To start things off, a builtin function could be provided such as def costart(f, *args, **kwds): return f.__cocall__(*args, **kwds) which would return an object that a coroutine driver could treat as a generator. I think this scheme would go a long way towards satisfying Antoine's desire to conceptually separate generators and coroutines. It would also enable an implementation to implement coroutines using a different mechanism from generators if it wanted to. -- Greg This email may be confidential and subject to legal privilege, it may not reflect the views of the University of Canterbury, and it is not guaranteed to be virus free. If you are not an intended recipient, please notify the sender immediately and erase all copies of the message and any attachments. Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more information.
On 02Aug2010 23:16, ghazel@gmail.com <ghazel@gmail.com> wrote: | On Mon, Aug 2, 2010 at 11:17 PM, Gregory Ewing | <greg.ewing@canterbury.ac.nz> wrote: | > Would you be interested in a system which requires marking | > calls to coroutines, but tells you immediately when you | > have forgotten to mark such a call? | > | > It might work something like this: | > | > 1. In a cofunction, a call to another cofunction must | > be marked with 'cocall', e,g. | > | > z = cocall f(x, y) | > | > 2. Cofunctions *cannot* be called normally -- they do | > not have a __call__ method, only a __cocall__ method. | > | > So if you try to call a cofunction without using cocall, | > you get an exception. If you try to call an ordinary function | > using cocall, you get an exception. If you try to use cocall | > but forget to declare the function with codef, you get an | > exception (because cocall would only be allowed inside a | > cofunction). | | I like this idea. Having just caught up with this thread, my first thought on reading the opening post was that it could do with (2), above. So a big +1 from me. It avoids misuse by failing early, and makes the failure reason obvious. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ ... you could spend *all day* customizing the title bar. Believe me. I speak from experience. - Matt Welsh
Guido van Rossum wrote:
That would cover part of it, the part about not being able to make a normal call to a cofunction. But it wouldn't enforce only being able to use cocall inside a cofunction, or remove the need for a dummy yield in a cofunction that doesn't otherwise have any. Also, if cocall is just a function, you would still have to use 'yield from' on the result, so all your coroutine calls would end up looking like yield from cocall(f, args) which makes them even more verbose and thumbstickingoutish. -- Greg
On Mon, Aug 2, 2010 at 12:39 PM, Guido van Rossum <guido@python.org> wrote:
We already use decorators to change interfaces so I am +1 on reusing what people already know. contextlib.contextmanager is shorthand for "the following function is shorthand for a contextmanager" from contextlib import contextmanager @contextmanager def myfunc(...): ... So the generator equivalent would be from somelib import generator @generator def myfunc(...): ... Where generator is as simple as def generator(func): def inner(*args, **opts): if False: yield return func(*args, **opts) return inner But the bulk of Greg's proposal is to transform the called function in one of two ways In the original: to make this return func(*args, **opts) equivalent to this yield from func.__cocall__(*args, **opts) # func must be defined with 'codef' or support __cocall__ Or in his second suggested form to make this cocall func(*args, **opts) equivalent to this yield from func.__cocall__(*args, **opts) # func must support __cocall__ I'm not sure if the "codef" keyword is included in the second form. I'm -1 on the first proposal because it buries that the calling function is a generator. "yield from" (which it would be a synonym or replacement for) lets you know the called function is a generator without having to read the body of the called function. I'm -1 on the 2nd form (explicit "cocall") because it is a synonym for "yield from" and "yield from" fits my brain better because reads as "this is yield-like but slightly different." -Jack
Jack Diederich wrote:
Yes, it is. An important part of it is that 'cocall' would only be allowed inside a function defined with 'codef'. Together with the other restrictions, this makes it impossible to mix coroutine and non-coroutine code in invalid ways. I don't think it's possible to get that using yield-from and decorators (at least not without a lot of inefficient hackery). -- Greg
participants (10)
-
Antoine Pitrou -
Bruce Leban -
Cameron Simpson -
ghazel＠gmail.com -
Greg Ewing -
Gregory Ewing
-
Guido van Rossum -
Jack Diederich -
Nick Coghlan -
Scott Dial