[Python-ideas] Revised**12 PEP on Yield-From

[Greg Ewing]

Erik Groeneveld wrote:

> I do not see how a yield-from without support for
> splitting boundaries would be combined with my own code to do the
> latter.

Seems to me you could yield a sequence of values
to be pushed back, and use a top-level driver
something like this:

   def pushingback(g):
     queue = []
     try:
       while 1:
         queue.append(yield)
         while queue:
           pushback = g.send(queue.pop(0))
           if pushback:
             queue.extend(pushback)
     except StopIteration, e:
       return e.value

> Well, the whole point of using coroutines is to avoid buffering.

Arranging processing as a pipeline avoids having to buffer
all of the intermediate data. But if lookahead is required,
then you need at least a small amount of buffering somewhere,
whether it's explicit or implicit.

You want to do the buffering implicitly, by reading ahead
and pushing back the stuff you don't want, making the
caller responsible for holding onto it.

While that's certainly possible, I'm far from convinced
it's the easiest or clearest way to go about it. I would
be more inclined to insert an explicit buffering object,
such as a queue, between the two stages of the pipeline.

> Thirdly, there seems to be some sort of unspoken 'protocol' with
> generators.  A next() is actually send(None) and vaguely means 'I want
> data'.  It the same vein 'x = yield' actually is 'x = yield None' and
> also vaguely means 'I want data'.  So the None seems to play a special
> role.

There is a one-to-one relationship between yields in
the generator and next/send calls by the caller. Each
yield provides an opportunity to transmit one value
and receive one value. If either of these opportunities
is unused, None is substituted. This is no more special
than a function returning None when it has no other value
of interest.

This lockstep execution between producer and consumer
is a feature of any coroutine system in which one
coroutine sends values directly to another, without
any form of buffering between.

This may be an unfamiliar concept to programmers these
days, because the modern incarnations of coroutines
normally encountered (OS-provided processes and threads)
*never* send values to each other directly -- there's
always some kind of IPC object in between, such as a
pipe or queue.

So when seeing the generator send/yield mechanism for
the first time, it's tempting to think of it as being
like sending data to a process through a pipe. But
it's not really like that -- it's a lower-level
facility.

> JSP requires a lookahead
> and the coroutines must have some way to support this.  (Introducing a
> stream or buffer would put us back to where we started of course).

I don't see why. You don't have to buffer all the data,
only the minimum required to support the amount of
lookahead needed. And I can't see why a pair of
coroutines communicating via an IPC object that
supports lookahead can't be just as simple and clear
as using some kind of pushback mechanism.

-- 
Greg