[Python-ideas] Possible PEP 380 tweak

[Guido van Rossum]

[Changed subject]

> On 2010-10-25 04:37, Guido van Rossum wrote:
>> This should not require threads.
>>
>> Here's a bare-bones sketch using generators:
[...]

On Mon, Oct 25, 2010 at 3:19 AM, Jacob Holm <jh at improva.dk> wrote:
> If you don't care about allowing the funcs to raise StopIteration, this
> can actually be simplified to:
[...]

Indeed, I realized this after posting. :-) I had several other ideas
for improvements, e.g. being able to pass an initial value to the
reduce-like function or even being able to supply a reduce-like
function of one's own.

> More interesting (to me at least) is that this is an excellent example
> of why I would like to see a version of PEP380 where "close" on a
> generator can return a value (AFAICT the version of PEP380 on
> http://www.python.org/dev/peps/pep-0380 is not up-to-date and does not
> mention this possibility, or even link to the heated discussion we had
> on python-ideas around march/april 2009).

Can you dig up the link here?

I recall that discussion but I don't recall a clear conclusion coming
from it -- just heated debate.

Based on my example I have to agree that returning a value from
close() would be nice. There is a little detail, how multiple
arguments to StopIteration should be interpreted, but that's not so
important if it's being raised by a return statement.

> Assuming that "close" on a reduce_collector generator instance returns
> the value of the StopIteration raised by the "return" statements, we can
> simplify the code even further:
>
>
> def reduce_collector(func):
>    try:
>        outcome = yield
>    except GeneratorExit:
>        return None
>    while True:
>        try:
>            val = yield
>        except GeneratorExit:
>            return outcome
>        outcome = func(outcome, val)
>
> def parallel_reduce(iterable, funcs):
>    collectors = [reduce_collector(func) for func in funcs]
>    for coll in collectors:
>        next(coll)
>    for val in iterable:
>        for coll in collectors:
>            coll.send(val)
>    return [coll.close() for coll in collectors]
>
>
> Yes, this is only saving a few lines, but I find it *much* more readable...

I totally agree that not having to call throw() and catch whatever it
bounces back is much nicer. (Now I wish there was a way to avoid the
"try..except GeneratorExit" construct in the generator, but I think I
should stop while I'm ahead. :-)

The interesting thing is that I've been dealing with generators used
as coroutines or tasks intensely on and off since July, and I haven't
had a single need for any of the three patterns that this example
happened to demonstrate:

- the need to "prime" the generator in a separate step
- throwing and catching GeneratorExit
- getting a value from close()

(I did have a lot of use for send(), throw(), and extracting a value
from StopIteration.)

In my context, generators are used to emulate concurrently running
tasks, and "yield" is always used to mean "block until this piece of
async I/O is complete, and wake me up with the result". This is
similar to the "classic" trampoline code found in PEP 342.

In fact, when I wrote the example for this thread, I fumbled a bit
because the use of generators there is different than I had been using
them (though it was no doubt thanks to having worked with them
intensely that I came up with the example quickly).

So, it is clear that generators are extremely versatile, and PEP 380
deserves several good use cases to explain all the API subtleties.

BTW, while I have you, what do you think of Greg's "cofunctions" proposal?

-- 
--Guido van Rossum (python.org/~guido)