<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Guido<br>
<br>
I like the way you are building the description up from the simple
case, but I think you are missing a few details along the way.<br>
Those details are what has been driving the discussion, so I think it
is important to get them handled. I'll comment on each point as I get
to it.<br>
<br>
<br>
Guido van Rossum wrote:<br>
<blockquote
cite="mid:ca471dc20904041329p31571e51o44d0ea8bcd74f96f@mail.gmail.com"
type="cite">I want to name the new exception ReturnFromGenerator to
minimize the similarity with GeneratorExit [...]<br>
</blockquote>
<br>
Fine with me, assuming we can't get rid of it altogether. <br>
<br>
[Snipped description of close() and __del__(), which I intend to
comment on in the other thread]<br>
<br>
<blockquote
cite="mid:ca471dc20904041329p31571e51o44d0ea8bcd74f96f@mail.gmail.com"
type="cite">[Guido]<br>
>> Oh, and "yield from" competes with @couroutine over<br>
>> when the initial next() call is made, which again suggests the
two<br>
>> styles (yield-from and coroutines) are incompatible.<br>
><br>
> It is a serious problem, because one of the major points of the
PEP is that<br>
> it should be useful for refactoring coroutines. As a matter of
fact, I<br>
> started another thread on this specific issue earlier today which
only Nick<br>
> has so far responded to. I think it is solvable, but requires
some more<br>
> work.<br>
<br>
I think that's the thread where I asked you and Nick to stop making
more proposals.I a worried that a solution would become too complex,
and I want to keep the "naive" interpretation of "yield from EXPR" to
be as close as possible to "for x in EXPR: yield x". I think the
@coroutine generator (whether built-in or not) or explicit "priming" by
a next() call is fine.<br>
</blockquote>
<br>
I think it is important to be able to use yield-from with a @coroutine,
but I'll wait a bit before I do more on that front (except for a few
more comments in this mail). There are plenty of other issues to
tackle.<br>
<br>
<br>
<blockquote
cite="mid:ca471dc20904041329p31571e51o44d0ea8bcd74f96f@mail.gmail.com"
type="cite">So now let me develop my full thoughts on yield-from. This
is unfortunately long, because I want to show some intermediate stages.
I am using a green font for new code. I am using stages, where each
stage provides a better approximation of the desired semantics. Note
that each stage *adds* some semantics for corner cases that weren't
handled the same way in the previous stage. Each stage proposes an
expansion for "RETVAL = yield from EXPR". I am using Py3k syntax.<br>
</blockquote>
[snip stage 1-3]<br>
<blockquote
cite="mid:ca471dc20904041329p31571e51o44d0ea8bcd74f96f@mail.gmail.com"
type="cite"> 4. Stage four adds handling for ReturnFromGenerator, in
both places
where next() is called:<br>
<br>
<span style="font-family: courier new,monospace;">it = iter(EXPR)</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">try:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
x = next(it)</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except
StopIteration:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
RETVAL = e.value</span><br style="color: rgb(0, 0, 0);">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">except
ReturnFromGenerator as e:</span><br
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
RETVAL = e.value; break</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">else:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
while True:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
yield x</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
try:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
x = next(it)</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
except StopIteration:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
RETVAL = None; break</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
except ReturnFromGenerator as e:</span><br
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
RETVAL = e.value; break</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> yield x</span><br>
</blockquote>
<br>
(There are two cut'n'paste errors here. The first "break" and the
second "yield x" shouldn't be there. Just wanted to point it out in
case this derivation makes it to the PEP)<br>
<br>
<blockquote
cite="mid:ca471dc20904041329p31571e51o44d0ea8bcd74f96f@mail.gmail.com"
type="cite"><br>
5. Stage five shows what should happen if "yield x" above returns a
value: it is passed into the subgenerator using send(). I am ignoring
for now what happens if it is not a generator; this will be cleared up
later. Note that the initial next() call does not change into a send()
call, because there is no value to send before before we have yielded:<br>
<br>
<span style="font-family: courier new,monospace;"></span></blockquote>
[snipped code for stage 5]<br>
<br>
The argument that we have no value to send before we have yielded is
wrong. The generator containing the "yield-from" could easily have a
value to send (or throw), and if iter(EXPR) returns a coroutine or a
non-generator it could easily be ready to accept it. That is the idea
behind my attempted fixes to the @coroutine issue.<br>
<br>
<blockquote
cite="mid:ca471dc20904041329p31571e51o44d0ea8bcd74f96f@mail.gmail.com"
type="cite">6. Stage six adds more refined semantics for when "yield
x" raises an exception: it is thrown into the generator, except if it
is GeneratorExit, in which case we close() the generator and re-raise
it (in this case the loop cannot continue so we do not set RETVAL):<br>
<br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"></span></blockquote>
[snipped code for stage 6]<br>
<br>
This is where the fun begins. In an earlier thread we concluded that
if the thrown exception is a StopIteration and the *same* StopIteration
instance escapes the throw() call, it should be reraised rather than
caught and turned into a RETVAL. The reasoning was the following
example:<br>
<br>
<pre>def inner():
for i in xrange(10):
yield i
def outer():
yield from inner()
print "if StopIteration is thrown in we shouldn't get here"
</pre>
<br>
Which we wanted to be equivalent to:<br>
<br>
<pre>def outer():
for i in xrange(10):
yield i
print "if StopIteration is thrown in we shouldn't get here"
</pre>
<br>
The same argument goes for ReturnFromGenerator, so the expansion at
this stage should be more like:<br>
<br>
<pre>it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = None
except ReturnFromGenerator as e:
RETVAL = e.value
else:
while True:
<font color="#009900"> try:
</font> v = yield x
<font color="#009900"> except GeneratorExit:
it.close()
raise
except BaseException as e:
try:
x = it.throw(e) # IIRC this includes the correct traceback in 3.x so we don't need to use sys.exc_info
except StopIteration as r:
if r is e:
raise
RETVAL = None; break
except ReturnFromGenerator as r:
if r is e:
raise
RETVAL = r.value; break
else:
</font> try:
x = it.send(v)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break</pre>
<br>
Next issue is that the value returned by it.close() is thrown away by
yield-from. Here is a silly example:<br>
<br>
<pre>def inner():
i = 0
while True
try:
yield
except GeneratorExit:
return i
i += 1
def outer():
try:
yield from inner()
except GeneratorExit:
# nothing I can write here will get me the value returned from inner()
</pre>
<br>
Also the trivial:<br>
<br>
<pre>def outer():
return yield from inner()
</pre>
<br>
Would swallow the return value as well.<br>
<br>
I have previously suggested attaching the return value to the
(re)raised GeneratorExit, and/or saving the return value on the
generator and making close return the value each time it is called. We
could also choose to define this as broken behavior and raise a
RuntimeError, although it seems a bit strange to have yield-from treat
it as an error when close doesn't. Silently having the yield-from
construct swallow the returned value is my least favored option.<br>
<br>
<blockquote
cite="mid:ca471dc20904041329p31571e51o44d0ea8bcd74f96f@mail.gmail.com"
type="cite"><br>
7. In stage 7 we finally ask ourselves what should happen if it is not
a generator (but some other iterator). The best answer seems subtle:
send() should degenerator to next(), and all exceptions should simply
be re-raised. We can conceptually specify this by simply re-using the
for-loop expansion:<br>
<br>
<span style="font-family: courier new,monospace;">it = iter(EXPR)</span><br
style="font-family: courier new,monospace;">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">if
<it is not a generator>:</span><br
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
for x in it:</span><br
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
yield next(x)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> RETVAL = None</span><br
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span
style="font-family: courier new,monospace; color: rgb(0, 153, 0);">else:</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> </span><span
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">try:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
x = next(it)</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace;"> </span><span
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except
StopIteration:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
RETVAL = None</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace;"> </span><span
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except
ReturnFromGenerator as e:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
RETVAL = e.value</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace;"> </span><span
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">else:</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> while True:</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
try:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
v = yield x<br>
except GeneratorExit:<br>
it.close()<br>
raise<br>
except:<br>
try:<br>
x = it.throw(*sys.exc_info())<br>
except StopIteration:<br>
RETVAL = None; break<br>
except ReturnFromGenerator as e:<br>
RETVAL = e.value; break<br>
else:<br>
try:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
x = it.send(v)</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> except
StopIteration:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> RETVAL =
None; break</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
except ReturnFromGenerator as e:</span><br
style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
RETVAL = e.value; break</span><br
style="font-family: courier new,monospace;">
<br>
Note: I don't mean that we literally should have a separate code path
for non-generators. But writing it this way adds the generator test to
one place in the spec, which helps understanding why I am choosing
these semantics. The entire code of stage 6 degenerates to stage 1 if
we make the following substitutions:<br>
<br>
<span style="font-family: courier new,monospace;">it.send(v)
-> next(v)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">it.throw(sys.exc_info())
-> raise</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">it.close()
-> pass</span><br>
<br>
(Except for some edge cases if the incoming exception is StopIteration
or ReturnFromgenerator, so we'd have to do the test before entering the
try/except block around the throw() or send() call.)<br>
<br>
We could do this based on the presence or absence of the
send/throw/close attributes: this would be duck typing. Or we could use
isinstance(it, types.GeneratorType). I'm not sure there are strong
arguments for either interpretation. The type check might be a little
faster. We could even check for an exact type, since GeneratorType is
final. Perhaps the most important consideration is that if EXPR
produces a file stream object (which has a close() method), it would
not consistently be closed: it would be closed if the outer generator
was closed before reaching the end, but not if the loop was allowed to
run until the end of the file. So I'm leaning towards only making the
generator-specific method calls if it is really a generator.<br>
</blockquote>
<br>
Like Greg, I am in favor of duck-typing this as closely as possible.
My preferred treatment for converting stage 6 to stage 7 goes like this:<br>
<br>
<pre>x = it.close() -->
m = getattr(it, 'close', None)
if m is not None:
x = it.close()
else:
x = None
x = it.send(v) -->
if v is None:
x = next(it)
else:
try:
m = it.send
except AttributeError:
m = getattr(it, 'close', None)
if m is not None:
it.close() # in this case I think it is ok to ignore the return value
raise
else:
x = m(v)
x = throw(e) -->
m = getattr(it, 'throw', None)
if m is not None:
x = m()
else:
m = getattr(it, 'close', None)
if m is not None:
it.close() # in this case I think it is ok to ignore the return value
raise e
</pre>
In this version it is easy enough to wrap the final iterator if you
want different behavior. With your version it becomes difficult to
replace a generator that is used in a yield-from with an iterator.
(You would have to wrap the iterator in a generator that mostly
consisted of the expansion from this PEP with the above substitution).<br>
<br>
I don't think we need to worry about performance at this stage. AFAICT
from the patch I was working on, the cost of a few extra checks is
negligible compared to the savings you get from using yield-from in the
first place.<br>
<br>
Best regards<br>
- Jacob<br>
<br>
</body>
</html>