Re: [Python-ideas] Revised revised revised PEP on yield-from
[Resend, hopefully bag.python.org is fixed again]
On Mon, Feb 16, 2009 at 1:52 PM, Greg Ewing
Guido van Rossum wrote:
(a) I don't know if the PEP proposes that "yield from expr" should return the last value returned by (i.e. sent to) a yield somewhere deeply nested; I think this would be useful.
No, it doesn't. Its value is the value passed to the 'return' statement that terminates the subgenerator (and generators are enhanced to permit return with a value).
My reason for doing this is so you can use subgenerators like functions in a generator that's being used as a lightweight thread.
There better be a pretty darn good reason to do this. I really don't like overloading return this way -- normally returning from a generator is equivalent to falling off the end and raises StopIteration, and I don't think you can change that easily.
(b) I hope the PEP also explains what to do if "expr" is not a generator but some other kind of iterator.
Yes, it does. Currently I'm proposing that, if the relevant methods are not defined, send() is treated like next(), and throw() and close() do what they would have done normally on the parent generator.
I'm not sure I like this interpretation of .send() -- it looks asymmetrical with the way .send() to non-generator iterators is treated in other contexts, where it is an error. I'm fine with the other two though, so I'm not sure how strong my opposition should be.
(c) A quick skim of the PEP didn't show suggestions for how to implement this.
One way would be to simply emit the bytecode corresponding to the presented expansion, although that wouldn't be very efficient in terms of either speed or code size.
Also pretty complex given the special cases for .send(), .throw() and .close() -- if it weren't for pass-through it could b a simplified for-loop (leaving out the variable assignment), but because of the pass-through it seems it would have to be ugly.
The PEP also sketches an optimised implementation in which the generator has a slot which refers to the generator being delegated to. Calls to next(), send(), throw() and close() are forwarded via this slot if it is nonempty.
And that could in turn be a generator with another such slot, right? Hopefully the testing for the presence of .throw, .send and .close could be done once at the start of the yield-from and represented as a set of flags (or perhaps the test could be delayed until the first time it's needed). I recommend that you produce a working implementation of this; who knows what other issues you might run into (including, whether your proposed interpretation of return from a generator above makes sense.
There will still be a small overhead involved in the delegation, but it's only a chain of C function calls instead of Python ones, which ought to be a big improvement.
Agreed.
It might be possible to reduce the overhead even further by following the chain of delegation pointers in a loop until reaching the end and then calling the end generator directly. It would be trickier to get right, though, because you'd have to be prepared to back up and try earlier generators in the face of StopIterations.
Well there you have a question that could be answered by trying to implement it. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
I think we can already create coroutines and lightweight tasks already without the need of the "yield from" syntax. The trick is to assume that when a coroutine yields an other coroutine, the behaviour is the same as using "yield from". The code somehow looks like this : # This is the actual coroutine we want to write @coroutine def func(): yield func2() # just like 'yield from func2()' yield 10 # That would be the return value in a coroutine # This is the code -probably wrong, and missing many things, but that gives the idea- of the coroutine library class Coroutine: def __init__(generator): self.generator = generator def run(self): for value in self.generator: if not isinstance(value, Coroutine): yield value else: # XXX: should also do all the proper checking as 'yield from' do for subvalue in value.run(): yield subvalue def coroutine(func): def ret(): return Coroutine(func) return ret The proposal could make things faster, and also slightly more coherent. But what I don't really like with it, is that when you start to write coroutines, you have to use yield every time you call an other coroutine, and it make the code full of yield statement ; the proposal if adopted would make it even worst. Cheers, Guillaume Chereau -- http://charlie137.blogspot.com/
On Mon, Feb 16, 2009 at 7:05 PM, Guillaume Chereau
But what I don't really like with it, is that when you start to write coroutines, you have to use yield every time you call an other coroutine, and it make the code full of yield statement ; the proposal if adopted would make it even worst.
I don't see how the proposal would make it *worse* (assuming that's what you meant, and "worst" was a typo). Coroutines make my head hurt more than metaclasses, but I see value in the proposal besides coroutines. Recursive generators are pretty calling and the "for x in A: yield x" idiom gets tiresome to read and write. Personally I care more about having to mentally parse it and realize "oh, it's a recursive generator" than about the extra typing it requires (only seven characters more :-). The mental parsing is a relatively big burden: the whole first line gives no indication that it's a recursive iterator, because "for x in A:" could start a loop doing something quite different. Your code-reading brain has to find the correspondence between the loop control variable of the for-loop header and the variable yielded in the loop body, see that nothing else is going on, and *then* it can draw the conclusion about the recursive generator. Once learned, "yield from" is a mental shortcut that saves your code-reading brain time and effort, which in turn helps understanding the larger picture of which this construct is a part -- you don't have to push anything onto your mental stack to see what's going on with "yield from". -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guillaume Chereau wrote:
But what I don't really like with it, is that when you start to write coroutines, you have to use yield every time you call an other coroutine,
Yes, but that's unavoidable as long as you're faking things with generators instead of using real threads, unless some other construct is introduced that's tantamount to 'yield' by another name -- and then you have to remember to use that.
and it make the code full of yield statement ; the proposal if adopted would make it even worst.
I don't see how it would be any worse. Your code at first glance looks incomprehensible to me -- how am I supposed to know that the first 'yield' is a blocking operation while the second one is returning a value? It relies on obscure conventions implemented by some kind of wrapper that you have to learn about. -- Greg
Guillaume Chereau wrote:
But what I don't really like with it, is that when you start to write coroutines, you have to use yield every time you call an other coroutine,
Yes, but that's unavoidable as long as you're faking things with generators instead of using real threads, unless some other construct is introduced that's tantamount to 'yield' by another name -- and then you have to remember to use that. Yes I though about this idea, like adding an attribute to a function to tell the interpreter to magically yield its result, but that totally break the mental parsing of the code (not talking about
On Tue, Feb 17, 2009 at 1:44 PM, Greg Ewing
and it make the code full of yield statement ; the proposal if adopted would make it even worst.
I don't see how it would be any worse. Your code at first glance looks incomprehensible to me -- how am I supposed to know that the first 'yield' is a blocking operation while the second one is returning a value? It relies on obscure conventions implemented by some kind of wrapper that you have to learn about.
I Agree, and I don't like it either. It was just the easer way I found to implement micro threads using generators. The proposal would indeed make things more logical, specially if we can use 'return' into the generators. The point I wanted to make was that then we need to write "yield from" every time we call a coroutine from an other one, that is probably a lot, and so made me unhappy about the syntax. In the context of a coroutine, 'yield from' means : "we start this other coroutine, and return to the current coroutine when it is done", and I was expecting the syntax to somehow express this idea. On the other hand, the other usage of "yield from" (to replace : "for x in a: yield x") is totally fine. I tried to think of some other keywords to suggest that would suite both usages, but couldn't find anything. Best regards, Guillaume
Guillaume Chereau wrote:
In the context of a coroutine, 'yield from' means : "we start this other coroutine, and return to the current coroutine when it is done", and I was expecting the syntax to somehow express this idea.
For what it's worth, I tend to feel the same way. I was originally going to call it 'call': y = call g(x) but Guido convinced me that 'call' is far too generic a word and doesn't convey any connection with generators at all. If anyone has any other suggestions, I'll gladly consider them. -- Greg
Reading the last few messages what came to mind was 'yield to'. In the
coroutine contexts it might be more appropriate thinking of yielding
execution 'to' the other generator. But in other contexts it seems more
appropriate to think of yielding values 'from' the other generator. Of
course it does both of these things. And 'yield from' makes more sense to
me.
It reminds me of a Honeymooners skit with Ralph and Ed arguing about the
doors between two rooms labeled 'in' and 'out'.
Ralph: "You are supposed to go in the door marked 'In.'"
Ed: "I wasn't going in that room. I was coming out of this room."
Ralph: "You were not going out of this room. You were going in that
room."
Ed: "How could I go into that room without coming out of this room?"
--- Bruce
On Tue, Feb 17, 2009 at 1:04 AM, Greg Ewing
Guillaume Chereau wrote:
In the context of a coroutine, 'yield from' means : "we start this
other coroutine, and return to the current coroutine when it is done", and I was expecting the syntax to somehow express this idea.
For what it's worth, I tend to feel the same way. I was originally going to call it 'call':
y = call g(x)
but Guido convinced me that 'call' is far too generic a word and doesn't convey any connection with generators at all.
If anyone has any other suggestions, I'll gladly consider them.
-- Greg
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
participants (4)
-
Bruce Leban -
Greg Ewing -
Guido van Rossum -
Guillaume Chereau