Re: [Python-Dev] PEP 380 (yield from a subgenerator) comments
At 09:24 PM 3/25/2009 -0700, Guido van Rossum wrote:
ISTR that the motivation for adding new syntax is that the best you can do using a trampoline library is still pretty cumbersome to use when you have to write a lot of tasks and subtasks, and when using tasks is just a tool for getting things done rather than an end goal in itself. I agree that the motivation and the comparison should be added to the PEP (perhaps moving the trampoline sample *implementation* to a reference or an appendix, since it is only the appearance of the trampoline-*using* code that matters).
In particular, it should explain why these choices are so costly as to justify new syntax and a complex implementation: * decorator clearly identifying intent to create a task vs. no indication of task-ness * "yield Return(value)" vs. "return value" * "result = yield subgenerator()" vs. "result = yield from subgenerator()" Especially since the first two differences arguably make the non-"yield from" code clearer, and the third makes it more compact.
with judicious use of decorators and helper classes you can get a reasonable approximation, and I think Twisted uses something like this, so it's not just theory. I think the best you can do without new syntax though is still pretty cumbersome and brittle, which is why I have encouraged your PEP.
On the "cumbersome" side, there are only three differences, as I've shown above... and one of them uses less syntax than the PEP. I'm not sure what you mean by brittle. Twisted and PEAK have both had generator-based tasks for ages, and have been used in production for years, even before generators had send()/throw() -- the addition of yield expressions made it possible to get rid of the one previous brittleness, where you needed to do things like: yield subgenerator(); result = resume() in order to pass values or exceptions in. Since send()/throw() was added, the need to call a function after each yield was eliminated, at least from PEAK and my newer Trellis library; haven't looked at Twisted's tasks in a while. I believe there are other generator-based task libraries available on PyPI, but can't comment on their robustness. (The existence of throw(), by the way, makes it possible to produce tracebacks that look *exactly* as if you had called a series of functions, rather than iterating over a bunch of generators. The sample code I gave should do this correctly, as it was created by yanking out working, tested code from my Trellis library, and cutting out all the Trellis-specific bits.) If someone can find any semantic differences between the code I posted and the yield-from proposal (apart from the absence of the "for x in y: yield x" part of the functionality), I'd like to know about it... and it should go in the PEP.
On Thu, Mar 26, 2009 at 10:07 AM, P.J. Eby <pje@telecommunity.com> wrote:
At 09:24 PM 3/25/2009 -0700, Guido van Rossum wrote:
ISTR that the motivation for adding new syntax is that the best you can do using a trampoline library is still pretty cumbersome to use when you have to write a lot of tasks and subtasks, and when using tasks is just a tool for getting things done rather than an end goal in itself. I agree that the motivation and the comparison should be added to the PEP (perhaps moving the trampoline sample *implementation* to a reference or an appendix, since it is only the appearance of the trampoline-*using* code that matters).
In particular, it should explain why these choices are so costly as to justify new syntax and a complex implementation:
* decorator clearly identifying intent to create a task vs. no indication of task-ness * "yield Return(value)" vs. "return value" * "result = yield subgenerator()" vs. "result = yield from subgenerator()"
Especially since the first two differences arguably make the non-"yield from" code clearer, and the third makes it more compact.
with judicious use of decorators and helper classes you can get a reasonable approximation, and I think Twisted uses something like this, so it's not just theory. I think the best you can do without new syntax though is still pretty cumbersome and brittle, which is why I have encouraged your PEP.
On the "cumbersome" side, there are only three differences, as I've shown above... and one of them uses less syntax than the PEP.
I wasn't talking about keystrokes so much as to having to remember to use thes incantations and understand what they do and what their limitations are.
I'm not sure what you mean by brittle. Twisted and PEAK have both had generator-based tasks for ages, and have been used in production for years, even before generators had send()/throw() -- the addition of yield expressions made it possible to get rid of the one previous brittleness, where you needed to do things like:
yield subgenerator(); result = resume()
in order to pass values or exceptions in. Since send()/throw() was added, the need to call a function after each yield was eliminated, at least from PEAK and my newer Trellis library; haven't looked at Twisted's tasks in a while. I believe there are other generator-based task libraries available on PyPI, but can't comment on their robustness.
(Well here is Greg's requested use case for .send(). :-) By brittle I meant again having to be aware of those details of the mechanism that exist because of syntactic limitations, e.g. accidentally writing "return X" instead of "yield Return(X)".
(The existence of throw(), by the way, makes it possible to produce tracebacks that look *exactly* as if you had called a series of functions, rather than iterating over a bunch of generators. The sample code I gave should do this correctly, as it was created by yanking out working, tested code from my Trellis library, and cutting out all the Trellis-specific bits.)
If someone can find any semantic differences between the code I posted and the yield-from proposal (apart from the absence of the "for x in y: yield x" part of the functionality), I'd like to know about it... and it should go in the PEP.
I'll leave this up to Greg. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
(Well here is Greg's requested use case for .send(). :-)
There was a complaint that my return-value-with-send example was too much of a coroutine scenario, so I was hoping to find something un-coroutine-like. But if coroutines are the main uses for send in the first place, then it seems my example is fine as it is. BTW, I've thought of an answer as to why one would want to write a parser using send(). In the non-send() version I had to make the scanner a global variable in order to avoid passing it around among all the parsing functions. The send() technique avoids having to do either of those things. -- Greg
Greg Ewing wrote:
Guido van Rossum wrote:
(Well here is Greg's requested use case for .send(). :-)
There was a complaint that my return-value-with-send example was too much of a coroutine scenario, so I was hoping to find something un-coroutine-like. But if coroutines are the main uses for send in the first place, then it seems my example is fine as it is.
BTW, I've thought of an answer as to why one would want to write a parser using send(). In the non-send() version I had to make the scanner a global variable in order to avoid passing it around among all the parsing functions. The send() technique avoids having to do either of those things.
Someone pointed out in my tutorial today that the circuits module makes heavy use of the generators' two-way communications capability. Maybe of you looked at the code you would find low-hanging fruit you could pick for sensible use cases. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/
P.J. Eby wrote:
In particular, it should explain why these choices are so costly as to justify new syntax and a complex implementation:
If avoiding trampolines was the only reason for yield-from, that mightn't be enough justification on its own. But it addresses several other use cases as well. The justification for yield-from is the total of all those.
* decorator clearly identifying intent to create a task vs. no indication of task-ness
Currently the only indication that a function is a generator rather than an ordinary function is the presence of a 'yield' buried somewhere in the body. Some people complained about that back when generators were being designed, but we seem to have gotten used to it. I don't anticipate any worse problem here. -- Greg
participants (4)
-
Greg Ewing -
Guido van Rossum -
P.J. Eby -
Steve Holden