[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]
Guido van Rossum
guido at python.org
Sat Apr 4 22:29:00 CEST 2009
[Answering somewhat out of order; new proposal developed at the end.]
On Fri, Apr 3, 2009 at 4:25 PM, Jacob Holm <jh at improva.dk> wrote:
> I am still trying to get a clear picture of what kind of mistakes you are
> trying to protect against.
> If it is people accidently writing return in a generator when they really
> mean yield, that is what I thought the proposal for an alternate syntax
was
> for. That sounds like a good idea to me, especially if we could also ban
or
> discourage the use of normal return. But the alternate syntax doesn't
have
> to mean a different exception.
I am leaning the other way now. New syntax for returning in a value is a
high-cost proposition. Instead, I think we can guard against most of the
same mistakes (mixing yield and return in a generator used as an iterator)
by using a different exception to pass the value. This would delay the
failure to runtime, but it would still fail loudly, which is good enough for
me.
I want to name the new exception ReturnFromGenerator to minimize the
similarity with GeneratorExit: if we had both GeneratorExit and
GeneratorReturn there would be endless questions on the newbie forums about
the differences between the two, and people might use the wrong one. Since
ReturnFromGenerator goes *out* of the generator and GeneratorExit goes *in*,
there really are no useful parallels, and similar names would cause
confusion.
> I could easily see "return value" as a separate PEP, except PEP 380
> provides one of the better reasons for its inclusion. It might be good to
> figure out how this feature should work by itself before complicating
things
> by integrating it in the yield-from semantics.
Here are my curent thoughts on this. When a generator returns, the return
statement is treated normally (whether or not it has a value) until the
frame is about to be left (i.e. after any finally-clauses have run). Then,
it is converted to StopIteration if there was no value or
ReturnFromGenerator if there was a value. I don't care which one is picked
for an explicit "return None" -- that should be decided by implementation
expediency. (E.g. if one requires adding new opcodes and one doesn't, I'd
pick the one that doesn't.)
Normal loops (for-loops, list comprehensions, other implied loops) only
catch StopIteration, so that returning a value is still wrong here. But some
other contexts treat ReturnFromGenerator similar as StopIteration except the
latter conveys None and the former conveys an explicit value. This applies
to yield-from as well as to explicit or implied closing of the generator
(close() or deallocation).
So g.close() returns the value (I think I earlier said I didn't like that --
I turned around on this one). It's pseudo-code is roughly:
def close(it):
try:
it.throw(GeneratorExit)
except (GeneratorExit, StopIteration):
return None
except ReturnFromGenerator as e: # This block is really the only new thing
return e.value
# Other exceptions are passed out unchanged
else:
# throw() yielded a value -- unchanged
raise RuntimeError(.....)
Deleting a generator is like closing and printing (!) a traceback (to
stderr) if close() raises an exception. A returned value it is just ignored.
Explicit pseudo-code without falling back to close():
def __del__(it):
try:
it.throw(GeneratorExit)
except (GeneratorExit, StopIteration, ReturnFromGenerator):
pass
except:
# Some other exception happened
<print traceback>
else:
# throw() yielded another value
<print traceback>
I have also worked out what I want yield-from to do, see end of this
message.
[Guido]
>> Oh, and "yield from" competes with @couroutine over
>> when the initial next() call is made, which again suggests the two
>> styles (yield-from and coroutines) are incompatible.
>
> It is a serious problem, because one of the major points of the PEP is
that
> it should be useful for refactoring coroutines. As a matter of fact, I
> started another thread on this specific issue earlier today which only
Nick
> has so far responded to. I think it is solvable, but requires some more
> work.
I think that's the thread where I asked you and Nick to stop making more
proposals.I a worried that a solution would become too complex, and I want
to keep the "naive" interpretation of "yield from EXPR" to be as close as
possible to "for x in EXPR: yield x". I think the @coroutine generator
(whether built-in or not) or explicit "priming" by a next() call is fine.
-----
So now let me develop my full thoughts on yield-from. This is unfortunately
long, because I want to show some intermediate stages. I am using a green
font for new code. I am using stages, where each stage provides a better
approximation of the desired semantics. Note that each stage *adds* some
semantics for corner cases that weren't handled the same way in the previous
stage. Each stage proposes an expansion for "RETVAL = yield from EXPR". I am
using Py3k syntax.
1. Stage one uses the for-loop equivalence:
for x in EXPR:
yield x
RETVAL = None
2. Stage two expands the for-loop into an explicit while-loop that has the
same meaning. It also sets RETVAL when breaking out of the loop. This
prepares for the subsequent stages. Note that we have an explicit iter(EXPR)
call here, since that is what a for-loop does:
it = iter(EXPR)
while True:
try:
x = next(it)
except StopIteration:
RETVAL = None; break
yield x
3. Stage three further rearranges stage 2 without making semantic changes,
Again this prepares for later stages:
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = e.value
else:
while True:
yield x
try:
x = next(x)
except StopIteration:
RETVAL = None; break
4. Stage four adds handling for ReturnFromGenerator, in both places where
next() is called:
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = e.value
except ReturnFromGenerator as e:
RETVAL = e.value; break
else:
while True:
yield x
try:
x = next(it)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
yield x
5. Stage five shows what should happen if "yield x" above returns a value:
it is passed into the subgenerator using send(). I am ignoring for now what
happens if it is not a generator; this will be cleared up later. Note that
the initial next() call does not change into a send() call, because there is
no value to send before before we have yielded:
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = None
except ReturnFromGenerator as e:
RETVAL = e.value
else:
while True:
v = yield x
try:
x = it.send(v)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
6. Stage six adds more refined semantics for when "yield x" raises an
exception: it is thrown into the generator, except if it is GeneratorExit,
in which case we close() the generator and re-raise it (in this case the
loop cannot continue so we do not set RETVAL):
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = None
except ReturnFromGenerator as e:
RETVAL = e.value
else:
while True:
try:
v = yield x
except GeneratorExit:
it.close()
raise
except:
try:
x = it.throw(*sys.exc_info())
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
else:
try:
x = it.send(v)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
7. In stage 7 we finally ask ourselves what should happen if it is not a
generator (but some other iterator). The best answer seems subtle: send()
should degenerator to next(), and all exceptions should simply be re-raised.
We can conceptually specify this by simply re-using the for-loop expansion:
it = iter(EXPR)
if <it is not a generator>:
for x in it:
yield next(x)
RETVAL = None
else:
try:
x = next(it)
except StopIteration:
RETVAL = None
except ReturnFromGenerator as e:
RETVAL = e.value
else:
while True:
try:
v = yield x
except GeneratorExit:
it.close()
raise
except:
try:
x = it.throw(*sys.exc_info())
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
else:
try:
x = it.send(v)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
Note: I don't mean that we literally should have a separate code path for
non-generators. But writing it this way adds the generator test to one place
in the spec, which helps understanding why I am choosing these semantics.
The entire code of stage 6 degenerates to stage 1 if we make the following
substitutions:
it.send(v) -> next(v)
it.throw(sys.exc_info()) -> raise
it.close() -> pass
(Except for some edge cases if the incoming exception is StopIteration or
ReturnFromgenerator, so we'd have to do the test before entering the
try/except block around the throw() or send() call.)
We could do this based on the presence or absence of the send/throw/close
attributes: this would be duck typing. Or we could use isinstance(it,
types.GeneratorType). I'm not sure there are strong arguments for either
interpretation. The type check might be a little faster. We could even check
for an exact type, since GeneratorType is final. Perhaps the most important
consideration is that if EXPR produces a file stream object (which has a
close() method), it would not consistently be closed: it would be closed if
the outer generator was closed before reaching the end, but not if the loop
was allowed to run until the end of the file. So I'm leaning towards only
making the generator-specific method calls if it is really a generator.
--
--Guido van Rossum (home page:
http://www.python.org/~guido/<http://www.python.org/%7Eguido/>
)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090404/0dad23ab/attachment.html>
More information about the Python-ideas
mailing list