[Python-ideas] yield-from practical experience

[Erik Groeneveld]

Andrew McNabb suggested I take a look at the python-ideas thread about
yield-from because I have written a little framework called Weightless
that supports decomposing a program into generators.  I am aware that
I am a bit late, but here is my two cents.

I read the PEP and the discussions (most of it, to be fair) and I
compared it to what I have been creating the last couple of years.
For one thing, it is certainly not easy to create a comprehensible way
of decomposing programs into generators, especially in the presence of
exceptions.  It took me two years to understand it, and I have coded
it into a single generator called 'compose'
(http://www.weightless.io/compose).

As some people wanted to see an implementation of 'yield from': here
is one that actually uses the normal yield and just assumes any
generator is to be descended into.
(http://weightless.svn.sourceforge.net/viewvc/weightless/trunk/weightless/_compose.py?revision=17&view=markup)
'compose' allows one to decompose a program into generators, just like
it has been proposed.

As was is also questioned why all the complexity is needed. I believe
it is necessary to create an clear, abstract, comprehensible and
consistent programming interface.  Consider these examples:

Many programmers would expect this to work:

       def myHandler():
               try:
                       request = yield from readRequest()
               except:
                       handle error

But it is really not working without catching exceptions from
readRequest() and forward them to myHandler().

Again, many programmers would expect this to work:


       def a():
               yield from b()
       def b():
               yield from c()
       def c():
               raise Exception('b')
               yield 1

You would like to see a stack trace like:

       Traceback (most recent call last):
         File "example.py", line 34, in
           list(compose(a()))
         File "example.py", line 27, in a
           yield b()
         File "example.py", line 29, in b
           yield c()
         File "example.py", line 32, in c
           raise Exception('b')
       Exception: b

but you will see this, for more of less the same reason as why you
weren't able to catch exceptions in the first place:

Traceback (most recent call last):
         File "probeer.py", line 34, in
           list(compose(a()))
         File "probeer.py", line 32, in c
           raise Exception('b')
       Exception: b

We succeeded to create a rather complicated 'compose' that give an
intuitive decomposition tool for programmers and we are using it at a
daily basis. (see link above)

One of the main advantages (besides lightweight threading) is the use
of JSP style pipelines.  Things like for exampe an HTTP protocol
handler are much easier (have a more friendly flow, as Klaas van
Schelven puts it) to create than with callbacks.  An simple example
can bee seen here:
http://weightless.svn.sourceforge.net/viewvc/weightless/trunk/examples/callbacksvsgenerator.py?revision=9&view=markup
 I've set this up for the Dutch Python Usergroup and it demonstrates
the idea.

I really would be glad if something like 'yield from' becomes part of
the language.  I would love to cooperate during the realization and
testing so I can validate my own bakery of the last years and perhaps
contribute something?


Erik Groeneveld