Erik Groeneveld wrote:

I do not see how a yield-from without support for splitting boundaries would be combined with my own code to do the latter.

Seems to me you could yield a sequence of values to be pushed back, and use a top-level driver something like this: def pushingback(g): queue = [] try: while 1: queue.append(yield) while queue: pushback = g.send(queue.pop(0)) if pushback: queue.extend(pushback) except StopIteration, e: return e.value

Well, the whole point of using coroutines is to avoid buffering.

Arranging processing as a pipeline avoids having to buffer all of the intermediate data. But if lookahead is required, then you need at least a small amount of buffering somewhere, whether it's explicit or implicit. You want to do the buffering implicitly, by reading ahead and pushing back the stuff you don't want, making the caller responsible for holding onto it. While that's certainly possible, I'm far from convinced it's the easiest or clearest way to go about it. I would be more inclined to insert an explicit buffering object, such as a queue, between the two stages of the pipeline.

Thirdly, there seems to be some sort of unspoken 'protocol' with generators. A next() is actually send(None) and vaguely means 'I want data'. It the same vein 'x = yield' actually is 'x = yield None' and also vaguely means 'I want data'. So the None seems to play a special role.

There is a one-to-one relationship between yields in the generator and next/send calls by the caller. Each yield provides an opportunity to transmit one value and receive one value. If either of these opportunities is unused, None is substituted. This is no more special than a function returning None when it has no other value of interest. This lockstep execution between producer and consumer is a feature of any coroutine system in which one coroutine sends values directly to another, without any form of buffering between. This may be an unfamiliar concept to programmers these days, because the modern incarnations of coroutines normally encountered (OS-provided processes and threads) *never* send values to each other directly -- there's always some kind of IPC object in between, such as a pipe or queue. So when seeing the generator send/yield mechanism for the first time, it's tempting to think of it as being like sending data to a process through a pipe. But it's not really like that -- it's a lower-level facility.

JSP requires a lookahead and the coroutines must have some way to support this. (Introducing a stream or buffer would put us back to where we started of course).

I don't see why. You don't have to buffer all the data, only the minimum required to support the amount of lookahead needed. And I can't see why a pair of coroutines communicating via an IPC object that supports lookahead can't be just as simple and clear as using some kind of pushback mechanism. -- Greg