yiled-from restrictions [was: x=(yield from) confusion]
On 4/9/09, Jacob Holm <jh@improva.dk> wrote:
Guido van Rossum wrote:
On Wed, Apr 8, 2009 at 7:52 PM, Jacob Holm <jh@improva.dk> wrote:
Does storing it as part of the frame object count as "naturally in the stack frame"?
Do you mean nothing user-visible, or do you really mean nothing, not even as an implementation detail?
That rules out Gregs and my patches as well. They both need extra state on the frame object to be able to implement yield-from in the first place.
Well, to do it efficiently anyhow... but it doesn't need to be user-visible, which is why I asked for clarification.
One final suggestion I have is to make yield-from raise a RuntimeError if used on a generator that already has a frame.
That would prevent priming the generator. It would also prevent "handled the header lines already; pass the data records off to a different routine." -jJ
Jim Jewett wrote:
Sure, if you change the patch to emit the same byte codes as the expansion in the PEP would, you can get by without. Of course that would increase the overhead of each delegated next(), send() or throw() call by at least one order of magnitude. (That is based on actual testing, where I compared my implementation of yield-from with a yield in a simple for-loop. Since the PEP expansion is more complex than a for-loop, I expect the overhead to be somewhat greater).
Yes, but since there is no sane way the this can work in the current proposal, I would rather raise a RuntimeError. - Jacob
Jacob Holm wrote:
Sure there is. E.g., suppose we have a simple file format with one header per line, separated from the main body by a line starting with "~". Header lines are processed on a per-line basis, but for the body, the unaltered lines are passed back to the client: HeaderFinished = object() def process_file(f): for line in f: if line.startswith('~'): break yield process_header(line) yield HeaderFinished yield from f For more complex cases, once caching of the methods in yield-from is explicitly disallowed (as per my other message), you can do something like:
So long as the PEP explicitly disallows caching of the bound methods (which I now think it should do) you can get as creative as you like by manipulating dynamically generated types at runtime. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
[deleted file processing example, but Greg may want to add it to the PEP as a clear motivating use case] On 4/9/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
I thought part of the reason to use a generator instead of an iterator class was speed/efficiency, which that constraint would seem to block. (name lookups at each level)
And if that becomes a standard idiom, was anything really simplified? -jJ
Jim Jewett wrote:
[deleted file processing example, but Greg may want to add it to the PEP as a clear motivating use case]
It isn't really a good example - it's one of the old use cases that is easily addressed by "for x in f: yield f" :)
Yes, because you only have to write start() once and then you can use it as many times as you like. The big gain from yield-from is to allow code containing a yield statement to be factored out correctly and relatively easily even when you want to use send() and throw(). My start() example is merely intended to show that even with a comparatively simplistic approach in the PEP that expects to be running a fresh iterator to exhaustion every time that yield-from is used, you can still support some fairly exotic use cases with low overhead by dynamically updating the subiterator's methods. The one change I am suggesting to Greg due to this is that we drop the idea of allowing the bound methods to be cached by the implementation. Since for loops don't cache the next() method, I no longer think the new expression should cache any of next(), send() or throw(). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Jim Jewett wrote:
Sure, if you change the patch to emit the same byte codes as the expansion in the PEP would, you can get by without. Of course that would increase the overhead of each delegated next(), send() or throw() call by at least one order of magnitude. (That is based on actual testing, where I compared my implementation of yield-from with a yield in a simple for-loop. Since the PEP expansion is more complex than a for-loop, I expect the overhead to be somewhat greater).
Yes, but since there is no sane way the this can work in the current proposal, I would rather raise a RuntimeError. - Jacob
Jacob Holm wrote:
Sure there is. E.g., suppose we have a simple file format with one header per line, separated from the main body by a line starting with "~". Header lines are processed on a per-line basis, but for the body, the unaltered lines are passed back to the client: HeaderFinished = object() def process_file(f): for line in f: if line.startswith('~'): break yield process_header(line) yield HeaderFinished yield from f For more complex cases, once caching of the methods in yield-from is explicitly disallowed (as per my other message), you can do something like:
So long as the PEP explicitly disallows caching of the bound methods (which I now think it should do) you can get as creative as you like by manipulating dynamically generated types at runtime. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
[deleted file processing example, but Greg may want to add it to the PEP as a clear motivating use case] On 4/9/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
I thought part of the reason to use a generator instead of an iterator class was speed/efficiency, which that constraint would seem to block. (name lookups at each level)
And if that becomes a standard idiom, was anything really simplified? -jJ
Jim Jewett wrote:
[deleted file processing example, but Greg may want to add it to the PEP as a clear motivating use case]
It isn't really a good example - it's one of the old use cases that is easily addressed by "for x in f: yield f" :)
Yes, because you only have to write start() once and then you can use it as many times as you like. The big gain from yield-from is to allow code containing a yield statement to be factored out correctly and relatively easily even when you want to use send() and throw(). My start() example is merely intended to show that even with a comparatively simplistic approach in the PEP that expects to be running a fresh iterator to exhaustion every time that yield-from is used, you can still support some fairly exotic use cases with low overhead by dynamically updating the subiterator's methods. The one change I am suggesting to Greg due to this is that we drop the idea of allowing the bound methods to be cached by the implementation. Since for loops don't cache the next() method, I no longer think the new expression should cache any of next(), send() or throw(). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (3)
-
Jacob Holm -
Jim Jewett -
Nick Coghlan