[Python-ideas] Possible PEP 380 tweak

[Ron Adam]

On 10/25/2010 10:13 AM, Guido van Rossum wrote:
> [Changed subject]
>
>> On 2010-10-25 04:37, Guido van Rossum wrote:
>>> This should not require threads.
>>>
>>> Here's a bare-bones sketch using generators:
> [...]
>
> On Mon, Oct 25, 2010 at 3:19 AM, Jacob Holm<jh at improva.dk>  wrote:
>> If you don't care about allowing the funcs to raise StopIteration, this
>> can actually be simplified to:
> [...]
>
> Indeed, I realized this after posting. :-) I had several other ideas
> for improvements, e.g. being able to pass an initial value to the
> reduce-like function or even being able to supply a reduce-like
> function of one's own.
>
>> More interesting (to me at least) is that this is an excellent example
>> of why I would like to see a version of PEP380 where "close" on a
>> generator can return a value (AFAICT the version of PEP380 on
>> http://www.python.org/dev/peps/pep-0380 is not up-to-date and does not
>> mention this possibility, or even link to the heated discussion we had
>> on python-ideas around march/april 2009).
>
> Can you dig up the link here?
>
> I recall that discussion but I don't recall a clear conclusion coming
> from it -- just heated debate.
>
> Based on my example I have to agree that returning a value from
> close() would be nice. There is a little detail, how multiple
> arguments to StopIteration should be interpreted, but that's not so
> important if it's being raised by a return statement.
>
>> Assuming that "close" on a reduce_collector generator instance returns
>> the value of the StopIteration raised by the "return" statements, we can
>> simplify the code even further:
>>
>>
>> def reduce_collector(func):
>>     try:
>>         outcome = yield
>>     except GeneratorExit:
>>         return None
>>     while True:
>>         try:
>>             val = yield
>>         except GeneratorExit:
>>             return outcome
>>         outcome = func(outcome, val)
>>
>> def parallel_reduce(iterable, funcs):
>>     collectors = [reduce_collector(func) for func in funcs]
>>     for coll in collectors:
>>         next(coll)
>>     for val in iterable:
>>         for coll in collectors:
>>             coll.send(val)
>>     return [coll.close() for coll in collectors]
>>
>>
>> Yes, this is only saving a few lines, but I find it *much* more readable...
>
> I totally agree that not having to call throw() and catch whatever it
> bounces back is much nicer. (Now I wish there was a way to avoid the
> "try..except GeneratorExit" construct in the generator, but I think I
> should stop while I'm ahead. :-)


This is how my mind wants to write this.

@consumer
def reduce_collector(func):
     try:
         value = yield            # No value to yield here.
         while True:
             value = func((yield), value)        # or here.
     except YieldError:
         # next was called not send.
         yield = value

def parallel_reduce(iterable, funcs):
     collectors = [reduce_collector(func) for func in funcs]
     for v in iterable:
         for coll in collectors:
             coll.send(v)
     return [next(c) for c in collectors]


It nicely separates input and output parts of a co-function, which can be 
tricky to get right when you have to receive and send at the same yield.

Maybe in Python 4k?  Oh well. :-)


> The interesting thing is that I've been dealing with generators used
> as coroutines or tasks intensely on and off since July, and I haven't
> had a single need for any of the three patterns that this example
> happened to demonstrate:
>
> - the need to "prime" the generator in a separate step

Having a consumer decorator would be good.

def consumer(f):
     @wraps(f)
     def wrapper(*args, **kwds):
         coroutine = f(*args, **kwds)
         next(coroutine)
         return coroutine
     return wrapper


Or maybe it would be possible for python to autostart a generator if it's 
sent a value before it's started?  Currently you get an almost useless 
TypeError.  The reason it's almost useless is unless you are testing for it 
right after you create the generator, you can't (easily) be sure it's not 
from someplace inside the generator.

Ron