Guido van Rossum wrote:

On Sun, Apr 5, 2009 at 7:46 AM, Jacob Holm <jh@improva.dk> wrote:

...

The argument that we have no value to send before we have yielded is wrong. The generator containing the "yield-from" could easily have a value to send (or throw), and if iter(EXPR) returns a coroutine or a non-generator it could easily be ready to accept it. That is the idea behind my attempted fixes to the @coroutine issue.

I think it's simpler to refrain from yield-from in that case and spell it out. If the value to send doesn't come from outside the outer generator, yield-from is not the solution.

But it *could* come from outside. If it is a coroutine calling another coroutine, it could have done any number of yields first, the last of which would return the value to be sent to the inner one. It bothers me a lot if you cannot use yield-from with coroutines, because most other uses I can see are just as easily written as for-loops. I'll think a bit more about this.

...

This is where the fun begins. In an earlier thread we concluded that if the thrown exception is a StopIteration and the *same* StopIteration instance escapes the throw() call, it should be reraised rather than caught and turned into a RETVAL. The reasoning was the following example:

def inner(): for i in xrange(10): yield i

def outer(): yield from inner() print "if StopIteration is thrown in we shouldn't get here"

Which we wanted to be equivalent to:

def outer(): for i in xrange(10): yield i print "if StopIteration is thrown in we shouldn't get here"

The same argument goes for ReturnFromGenerator, so the expansion at this stage should be more like:

[snip]

This example and reasoning are invalid. You shouldn't be throwing StopIteration (or ReturnFromGenerator) *into* a generator. That's something that should only come *out*.

I am not claiming that you *should* be throwing StopIteration to a generator, just that there is nothing that prevents you from doing it, so we need to consider what should happen if you do. The above reasoning based on the refactoring principle lead to one choice, which I happen to like. If you only focus on getting the expansion in the PEP as simple as possible you will probably make another choice. Note that if you don't handle StopIteration this way but just treat it as a normal StopIteration you open up for interesting ways to abuse yield-from, exactly by throwing StopIteration. In particular, if you use an iterator without throw or close methods you can break out of the innermost yield-from and even set the value to be returned. I don't mind either way. I just thought I would mention this in case you missed it. [snip my examples where the return value was swallowed]

...

I have previously suggested attaching the return value to the (re)raised GeneratorExit, and/or saving the return value on the generator and making close return the value each time it is called. We could also choose to define this as broken behavior and raise a RuntimeError, although it seems a bit strange to have yield-from treat it as an error when close doesn't. Silently having the yield-from construct swallow the returned value is my least favored option.

Attaching it to the GeneratorExit is just plain wrong -- this is an exception you throw *in*, not something that is thrown out (except when it bounces back).

I expanded a little bit on the idea in my reply to Greg that must have crossed your mail. Listed a number of possible solutions that had been discussed in my order of preference. I don't see a problem in having the language construct "yield-from" raise GeneratorExit with a value as a result of GeneratorExit.

One solution is not to use yield-from but write it out using yield and send (just like the full expansion, but you can probably drop most of the complexity for any particular example).

Another solution is not to use close() and GeneratorExit but some application-specific exception to signal the end.

That doesn't really solve the issue of what should happen if you write such code. What bothers me most is that the return value is silently swallowed.

But perhaps it would be okay to change the GeneratorExit handler in the expansion so that it passes through the return value with a StopIteration exception:

rv = it.close() if rv is None: raise StopIteration(rv) # Or ReturnFromGenerator(rv) else: raise

Alternatively, simpler:

it.throw(GeneratorExit) # We only get here if it yielded a value raise RuntimeError(...)

(Though this isn't exactly if we were to use duck typing.)

We could then write the first version of outer() like this:

def outer(): try: yield from inner() except StopIteration as e: ...access return value as e.value...

and I think the second (trivial) outer() will return inner()'s return value just fine, since it just passes through as a StopIteration value.

I don't mind catching an exception to get the value in this case. I just think it should be GeneratorExit i should catch. This is related to the question of what should happen if you throw StopIteration. If you don't special-case StopIteration in throw, using StopIteration for this is fine.

FWIW, I'm beginning to think that ReturnFromGenerator is a bit of a nuisance, and that it's actually fine to allow "return value" inside a generator to mean "raise StopIteration(value)" (well not quite at that point in the code but once we are about to clean up the frame). Maybe I've overstated the case for preventing beginners' mistakes. After all they'll notice that their generator returns prematurely when they include any kind of return value. Also if the StopIteration ends being printed as a traceback the value will be printed, which is the kind of hint newbies love.

Ok, then I will stop worrying about ReturnFromGenerator until it is brought up again. Best regards - Jacob

Guido van Rossum wrote:

On Sun, Apr 5, 2009 at 7:46 AM, Jacob Holm <jh@improva.dk> wrote:

...

The argument that we have no value to send before we have yielded is wrong. The generator containing the "yield-from" could easily have a value to send (or throw), and if iter(EXPR) returns a coroutine or a non-generator it could easily be ready to accept it. That is the idea behind my attempted fixes to the @coroutine issue.

I think it's simpler to refrain from yield-from in that case and spell it out. If the value to send doesn't come from outside the outer generator, yield-from is not the solution.

But it *could* come from outside. If it is a coroutine calling another coroutine, it could have done any number of yields first, the last of which would return the value to be sent to the inner one. It bothers me a lot if you cannot use yield-from with coroutines, because most other uses I can see are just as easily written as for-loops. I'll think a bit more about this.

...

This is where the fun begins. In an earlier thread we concluded that if the thrown exception is a StopIteration and the *same* StopIteration instance escapes the throw() call, it should be reraised rather than caught and turned into a RETVAL. The reasoning was the following example:

def inner(): for i in xrange(10): yield i

def outer(): yield from inner() print "if StopIteration is thrown in we shouldn't get here"

Which we wanted to be equivalent to:

def outer(): for i in xrange(10): yield i print "if StopIteration is thrown in we shouldn't get here"

The same argument goes for ReturnFromGenerator, so the expansion at this stage should be more like:

[snip]

This example and reasoning are invalid. You shouldn't be throwing StopIteration (or ReturnFromGenerator) *into* a generator. That's something that should only come *out*.

I am not claiming that you *should* be throwing StopIteration to a generator, just that there is nothing that prevents you from doing it, so we need to consider what should happen if you do. The above reasoning based on the refactoring principle lead to one choice, which I happen to like. If you only focus on getting the expansion in the PEP as simple as possible you will probably make another choice. Note that if you don't handle StopIteration this way but just treat it as a normal StopIteration you open up for interesting ways to abuse yield-from, exactly by throwing StopIteration. In particular, if you use an iterator without throw or close methods you can break out of the innermost yield-from and even set the value to be returned. I don't mind either way. I just thought I would mention this in case you missed it. [snip my examples where the return value was swallowed]

...

I have previously suggested attaching the return value to the (re)raised GeneratorExit, and/or saving the return value on the generator and making close return the value each time it is called. We could also choose to define this as broken behavior and raise a RuntimeError, although it seems a bit strange to have yield-from treat it as an error when close doesn't. Silently having the yield-from construct swallow the returned value is my least favored option.

Attaching it to the GeneratorExit is just plain wrong -- this is an exception you throw *in*, not something that is thrown out (except when it bounces back).

I expanded a little bit on the idea in my reply to Greg that must have crossed your mail. Listed a number of possible solutions that had been discussed in my order of preference. I don't see a problem in having the language construct "yield-from" raise GeneratorExit with a value as a result of GeneratorExit.

One solution is not to use yield-from but write it out using yield and send (just like the full expansion, but you can probably drop most of the complexity for any particular example).

Another solution is not to use close() and GeneratorExit but some application-specific exception to signal the end.

That doesn't really solve the issue of what should happen if you write such code. What bothers me most is that the return value is silently swallowed.

But perhaps it would be okay to change the GeneratorExit handler in the expansion so that it passes through the return value with a StopIteration exception:

rv = it.close() if rv is None: raise StopIteration(rv) # Or ReturnFromGenerator(rv) else: raise

Alternatively, simpler:

it.throw(GeneratorExit) # We only get here if it yielded a value raise RuntimeError(...)

(Though this isn't exactly if we were to use duck typing.)

We could then write the first version of outer() like this:

def outer(): try: yield from inner() except StopIteration as e: ...access return value as e.value...

and I think the second (trivial) outer() will return inner()'s return value just fine, since it just passes through as a StopIteration value.

I don't mind catching an exception to get the value in this case. I just think it should be GeneratorExit i should catch. This is related to the question of what should happen if you throw StopIteration. If you don't special-case StopIteration in throw, using StopIteration for this is fine.

FWIW, I'm beginning to think that ReturnFromGenerator is a bit of a nuisance, and that it's actually fine to allow "return value" inside a generator to mean "raise StopIteration(value)" (well not quite at that point in the code but once we are about to clean up the frame). Maybe I've overstated the case for preventing beginners' mistakes. After all they'll notice that their generator returns prematurely when they include any kind of return value. Also if the StopIteration ends being printed as a traceback the value will be printed, which is the kind of hint newbies love.

Ok, then I will stop worrying about ReturnFromGenerator until it is brought up again. Best regards - Jacob