[Python-ideas] PEP 380 alternative: A yielding function

[Guido van Rossum]

Have you really thought through the implementation in CPython? When a
non-generator function is called, there's usually a C stack frame in
the way which prevents yielding.

--Guido

On Tue, Jul 27, 2010 at 6:18 PM, Anders J. Munch <2010 at jmunch.dk> wrote:
> Looking at PEP 380 (http://www.python.org/dev/peps/pep-0380/), the
> need for yield forwarding syntax comes from the inability to delegate
> yielding functionality in the usual way. For example, if you have a
> recurring pattern including yields, like this (this is a toy example,
> please don't take it for more than that):
>
>  if a:
>      yield a
>  if b:
>      yield b
>
> you cannot do the extract function refactoring in the usual way:
>
>     def yield_if_true(x):
>      if x:
>          yield x
>  yield_if_true(a)
>  yield_if_true(b)
>
> because yield_if_true would become a generator.
>
> PEP 380 addresses this by making the workaround - "for x in
> yield_if_true(a): yield x" - easier to write.
>
> But suppose you could address the source instead?  Suppose you could
> write yield_if_true in such a way that it did not become a generator
> despite yielding?
>
> Syntactically this could be done with a yielding *function* in
> addition to the yield statement/expression, either as a builtin or a
> function in the sys module.  Let's call it 'yield_' , for lack of a
> better name.  The function would yield the nearest generator on the
> call stack.
>
> Now the example would work with a slight modifiction:
>
>  def yield_if_true(x):
>      if x:
>          yield_(x)
>  yield_if_true(a)
>  yield_if_true(b)
>
> The real benefits from a yield_ function come with recursive
> functions.  A recursive tree traversal that yields from the leaf nodes
> currently suffers from a performance penalty: Every yield is repeated
> as many times as the depth of the tree, turning a O(n) traversal
> algorithm into an O(n lg(n)) algorithm.  PEP 380 does not change that.
>
> But a yield_ function could be O(1), regardless of the forwarding
> depth.
>
> To be fair, a clever implementation might be able to short-circuit a
> 'yield from' chain and achieve the same thing.
>
> Two main drawbacks of yield_:
> - Difficulty of implementation. Generators would need to keep an
>  entire stack branch alive, instead of merely a single frame, and if
>  that somehow affects the efficiency of simple generators, that would
>  be bad.
> - 'the nearest generator on the call stack' is sort of a dynamic
>  scoping thing, which has its problems.  For example, if you forget
>  to make the relevant function a generator (the "if 0: yield None"
>  trick might have been needed but was forgotten), then the yield
>  would trickle up to some random generator higher up, with confusing
>  results.  And if you use yield_ in a callback, well, let's just say
>  that an interesting case too.
>
> All the same, if a yield_ function is practical, I think it is a
> better way to address the problems that motivate PEP 380.  I'm guessing you
> could implement 'yield from' as a pure-Python
> function using yield_, making yield_ strictly more powerful, although
> I couldn't say for sure as I haven't studied the enhanced generator
> protocol.
>
> regards, Anders
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
--Guido van Rossum (python.org/~guido)