[Python-ideas] PEP on yield-from: throw example

[Bruce Frederiksen]

Steven D'Aprano wrote:
> If I've understood the protoPEP, it wraps four distinct pieces of 
> functionality:
>
> "yield from" pass-through
> pass-through for send
> pass-through for throw
> pass-through for close
>
> I think each one needs to be justified, or at least explained, 
> individually. I'm afraid I'm not even clear on what pass-through for 
> send/throw/close would even mean, let alone why they would be useful. 
> Basic yield pass-through is obvious, and even if we decide that it's 
> nothing more than syntactic sugar for "for x in gen: yield x", I think 
> it's a clear win for readability. But the rest needs some clear, 
> simple examples of how they would be used.
OK, let's give this a try.  I to do several posts, one on each item 
above in an attempt to demonstrate what we're talking about here.

First of all, to be clear on this, the send, throw and close mechanisms 
were proposed in PEP 342 and adopted in Python 2.5.  For some reason 
though, these new mechanisms didn't seem to make it into the standard 
Python documentation.  So you'll need to read PEP 342 if you have any 
question on how these work.

This post is on "pass-through for close".

I've tried to make these as simple as possible, but there's still a 
little bit to it, so please bear with me.

Let's get started.

We're going to do a little loan application program.  We're going to 
process a list of loan applications.  Each loan application consists of 
a list of people.  If any of the people on the list qualify, then they 
get the loan.  If none of the people qualify, they don't get the loan.

We're going to have a generator that generates the individual names.  If 
the name does not qualify, then DoesntQualify is raised by the caller 
using the throw method:

class DoesntQualify(Exception): pass

Names = [['Raymond'], ['Bruce', 'Marilyn'], ['Jack', 'Jill']]

def gen(l):
    count = 0
    try:
        for names in l:
            count += 1
            for name in names:
                try:
                    yield name
                    break
                except DoesntQualify:
                    pass
            else:
                print names, "don't qualify"
    finally:
        print "processed", count, "applications"

Now we need a function that gets passed this generator and checks each 
name to see if it qualifies.  I would expect to be able to write:

def process(generator):
    for name in generator:
        if len(name) > 5:
            print name, "qualifies"
        else:
            raise DoesntQualify

But running this gives:

 >>> g = gen(Names)
 >>> process(g)
Raymond qualifies
Traceback (most recent call last):
  File "throw2.py", line 34, in <module>
    process(g)
  File "throw2.py", line 31, in process
    raise DoesntQualify
__main__.DoesntQualify

What I expected was the for statement in process would forward the 
DoesntQualify exception to the generator.  But it doesn't do this, so 
I'm left to do it myself.  My next try developing this example, was:

def process(generator):
    for name in generator:
        while True:
            if len(name) > 5:
                print name, "qualifies"
                break
            else:
                name = generator.throw(DoesntQualify)

But running this gives:

Raymond qualifies
Marilyn qualifies
['Jack', 'Jill'] don't qualify
processed 3 applications
Traceback (most recent call last):
  File "throw2.py", line 46, in <module>
    process2(gen(Names))
  File "throw2.py", line 43, in process2
    name = iterable.throw(DoesntQualify)
StopIteration

Oops, the final throw raised StopIteration when it hit the end of 
Names.  So I end up with:

def process(generator):
    try:
        for name in generator:
            while True:
                if len(name) > 5:
                    print name, "qualifies"
                    break
                else:
                    name = generator.throw(DoesntQualify)
    except StopIteration:
        pass

This one works:

Raymond qualifies
Marilyn qualifies
['Jack', 'Jill'] don't qualify
processed 3 applications

But by this time, it's probably more clear if I just abandon the for 
statement entirely:

def process(generator):
    name = generator.next()
    while True:
        try:
            if len(name) > 5:
                print name, "qualifies"
                name = generator.next()
            else:
                name = generator.throw(DoesntQualify)
        except StopIteration:
            break

But now I need to change process to add a limit to the number of 
accepted applications:

def process(generator, limit):
    name = generator.next()
    count = 1
    while count <= limit:
        try:
            if len(name) > 5:
                print name, "qualifies"
                name = generator.next()
                count += 1
            else:
                name = generator.throw(DoesntQualify)
        except StopIteration:
            break

Seems easy enough, except that this is broken again because the final 
"processed N applications" message won't come out if the limit is hit 
(unless you are running CPython and call it in such a way that the 
generator is immediately collected -- but this doesn't work on jython or 
ironpython).  That's what the close method is for, and I forgot to call it:

def process(generator, limit):
    name = generator.next()
    count = 1
    while count <= limit:
        try:
            if len(name) > 5:
                print name, "qualifies"
                name = generator.next()
                count += 1
            else:
                name = generator.throw(DoesntQualify)
        except StopIteration:
            break
     generator.close()

So what starts out conceptually simple, ends up more complicated and 
error prone that I had expected; and the reason is that the for 
statement doesn't support these new generators methods.  If it did, I 
would have:

def process(generator, limit):
    count = 1
    for generator as name:     # new syntax doesn't break old code
        if len(name) > 5:
            print name, "qualifies"
            count += 1
            if count > limit: break
        else:
            raise DoesntQualify   # new for passes this to generator.throw
    # new for remembers to call generator.close for me.

Now, we need to extend this because there are several lists of 
applications.  I'd like to be able to use the same gen function on each 
list, and the same process function and just introduce an intermediate 
generator that gathers up the output of several generators.  This is 
exactly what itertools.chain does!  So this should be very easy:

 >>> g1 = gen(Names1)
 >>> g2 = gen(Names2)
 >>> g3 = gen(Names3)
 >>> process(itertools.chain(g1, g2, g3), limit=5)

But, nope, itertools.chain doesn't honor the extra generator methods 
either.  If we had yield from, then I could use that instead of 
itertools.chain:

def multi_gen(gen_list):
    for gen in gen_list:
        yield from gen

When I use yield from, it sets multi_gen aside and lets process talk 
directly to each generator.  So I would expect that not only would 
objects yielded by each generator be passed directly back to process, 
but that exceptions passed in by process with throw would be passed 
directly to the generator.  Why would this *not* be the case?  With the 
for statement, I can see that doing the throw/close processing might 
break some legacy code and understand the reservation in doing so 
there.  But here we have a new language construct where we don't need to 
worry about legacy code.  It's also a construct dealing directly and 
exclusively with generators.

If I can't use yield from, and itertools.chain does work, and the for 
statement doesn't work, then I'm faced once again with having to code 
everything again myself:

def multi_gen(gen_list):
    for gen in gen_list:
        while True:
            try:
                yield gen.next()
            except DoesntQualify, e:
                yield gen.throw(e)
            except StopIteration:
                gen.close()

Yuck!  Did I get this one right?  Nope, same StopIteration problem with 
gen.throw...  Let's try:

def multi_gen(gen_list):
    for gen in gen_list:
        try:
            while True:
                try:
                    yield gen.next()
                except DoesntQualify, e:
                    yield gen.throw(e)
        except StopIteration:
            pass
        finally:
            gen.close()

Even more yuck!  This feels more like programming in assembler than 
python :-(

-bruce frederiksen