[Python-ideas] Revised**12 PEP on Yield-From

[Erik Groeneveld]

Greg,

I am still busy understanding what your PEP means to the framework
that I have been building.  I believe that, for practical usage, there
is still something missing, or at least, not clear.

Suppose we would like to write a HTTP protocol stack using generators,
and we have something like (actual working code, read 'yield from' for
every 'yield'):

   try:
           reqArgs = yield readRe(REGEXP.REQUEST, MAXREQUESTSIZE)
   except OverflowError:
       yield requestEntityTooLarge()
       yield HTTP.CRLF
       return
   headers = parseHeaders(reqArgs)
   yield processBody(headers)

The function 'readRe' will read and buffer a part of the request as
defined by the regular expression REGEXP.REQUEST, and it will raise
OverflowError when it keeps reading while never making a match.

The point is that readRe accepts chunks of data that are not aligned
to protocol boundaries.  This is a typical boundary clash as Jackson
calls it (I tend to think of this stuff as JSP pipelines) and JSP
describes how to solve it.

But to be able to solve it, the readRe generator must be able to
indicate that it has superfluous data, and this data must be processed
by other generators.  In the case of the example, 'readRe' might have
been reading parts of the body (assuming a POST request).

After I created 'compose' I started implementing practical stuff like
this, and it soon turned out that 'compose' must support boundary
clashes or all but toy problems would still be unsolvable with
generators.

Therefor 'compose' now has a feature to return a value together with
remaining data:

   raise StopIteration(retval, remainingChunk1, remainingChunk2, ...)

This will push retval to the delegating generator as a return value
for yield, and then feed the remainingChunk's to whatever generators
come next.  In your PEP, this would be a return statement of course.

Have you though about this?  How would you solve it?


Best regards,
Erik

E.J. Groeneveld
Seek You Too