[Python-ideas] Revised**7 PEP on Yield-From

[Jacob Holm]

Greg Ewing wrote:
> Jacob Holm wrote:
>> the expansion in your PEP actually has the behaviour you expect for 
>> the GeneratorExit example because GeneratorExit doesn't inherit from 
>> Exception.
>
> That's an accident, though, and it's possible I should have
> specified BaseException there. I still consider the explanation
> I gave to be the true one.
In that case, I think a clarification in the PEP would be in order.   I 
like the fact that the PEP-342 description of close does the right thing 
though.  If you want BaseException instead of Exception in the PEP, 
maybe you could replace the:

except Exception, _e:


line with:

except GeneratorExit:
    raise
except BaseException, _e:


This would make it clearer that the behavior of close is intentional, 
and would still allow delegating the throw of any exception not 
inheriting from GeneratorExit to the subiterator.

>
>> The other mismatch, concerning the missing "close" calls to the 
>> iterator, I still believe to be an issue.
>
> Can you elaborate on that? I thought a first you were expecting
> the implicit close of the generator that happens before it's
> deallocated to be passed on to the subiterator, but some of your
> examples seem to have the close happening *before* the del gen,
> so I'm confused.
>
Yes, I can see that the use of implicit close in that example was a 
mistake, and that I should have added a few more output lines to clarify 
the intent.  The close is definitely intended to happen before the del 
in the examples.  I have a better example here, with inline comments 
explaining what I think should happen at critical points (and why):

class iterator(object):
    """Simple iterator that counts to n while writing what is done to it"""

    def __init__(self, n):
        self.ctr = iter(xrange(n))

    def __iter__(self):
        return self

    def close(self):
        print "Close"

    def next(self):
        print "Next"
        return self.ctr.next()

    # no send method!

    # no throw method!


def generator(n):
    try:
        print "Getting first value from iterator"
        result = yield from iterator(n)
        print "Iterator returned", result
    finally:
        print "Generator closing"


g = generator(1)
g.next()
try:
     print "Calling g.next()"
     # This causes a StopIteration in iterator.next().  After grabbing
     # the value in the "except StopIteration" clause of the PEP
     # expansion, the "finally" clause calls iterator.close().  Any
     # other exception raised by next (or by send or throw if the
     # iterator had those) would also be handled by the finally
     # clause.  For well-behaved iterators, these calls to close would
     # be no-ops, but they are still required by the PEP as written.
     g.next()
except Exception, e:
    print type(e)
else:
    print 'No exception'
# This close should be a no-op. The exception we just caught should
# have already closed the generator.
g.close()
print '--'


g = generator(1)
g.next()
try:
    print "Calling g.send(42)"
    # This causes an AttributeError when looking up the "send" method.
    # The finally clause from the PEP expansion makes sure
    # iterator.close() is called.  This call is *not* expected to be a
    # no-op.
    g.send(42)
except Exception, e:
    print type(e)
else:
    print 'No exception'
# This close should be a no-op. The exception we just caught should
# have already closed the generator.
g.close()
print '--'


g = generator(1)
g.next()
try:
    print "Calling g.throw(ValueError)"
    # Since iterator does not have a "throw" method, the ValueError is
    # raised directly in the yield-from expansion in the generator.
    # The finally clause ensures that iterator.close() is called.
    # This call is *not* expected to be a no-op.
    g.throw(ValueError)
except Exception, e:
    print type(e)
else:
    print 'No exception'
# This close should be a no-op. The exception we just caught should
# have already closed the generator.
g.close()
print '--'


g = generator(1)
g.next()
try:
    print "Calling g.throw(StopIteration(42))"
    # The iterator still does not have a "throw" method, so the
    # StopIteration is raised directly in the yield-from expansion.
    # Then the exception is caught and converted to a value for the
    # yield-from expression.  Before the generator sees the value, the
    # finally clause makes sure that iterator.close() is called.  This
    # call is *not* expected to be a no-op.
    g.throw(StopIteration(42))
except Exception, e:
    print type(e)
else:
    print 'No exception'
# This close should be a no-op. The exception we just caught should
# have already closed the generator.
g.close()
print '--'



There is really four examples here.  The first one is essentially the 
same as last time, I just expanded the output a bit.  The next two 
examples are corner cases where the missing close makes a real 
difference, even for well-behaved iterators (this is not the case in the 
first example).  The fourth example catches a bug in the current version 
of my patch, and shows a potentially interesting use of an iterator 
without a send method in a yield-from expression.  The issue i have with 
your patch is that iterator.close() is not called in any of the four 
examples, even though my reading of the PEP suggests it should be. (I 
have confirmed that my reading matches the PEP by manually replacing the 
yield-from in the generator with the expansion from the PEP, just to be 
sure...)

The expected output is:

Getting first value from iterator
Next
Calling g.next()
Next
Close
Iterator returned None
Generator closing
<type 'exceptions.StopIteration'>
--
Getting first value from iterator
Next
Calling g.send(42)
Close
Generator closing
<type 'exceptions.AttributeError'>
--
Getting first value from iterator
Next
Calling g.throw(ValueError)
Close
Generator closing
<type 'exceptions.ValueError'>
--
Getting first value from iterator
Next
Calling g.throw(StopIteration(42))
Close
Iterator returned 42
Generator closing
<type 'exceptions.StopIteration'>
--