[Python-ideas] Possible PEP 380 tweak

[Guido van Rossum]

On Mon, Oct 25, 2010 at 6:01 PM, Ron Adam <rrr at ronadam.com> wrote:
>
> On 10/25/2010 03:21 PM, Guido van Rossum wrote:
>>
>> On Mon, Oct 25, 2010 at 12:53 PM, Ron Adam<rrr at ronadam.com>  wrote:
>>>
>>> This is how my mind wants to write this.
>>>
>>> @consumer
>>> def reduce_collector(func):
>>>    try:
>>>        value = yield            # No value to yield here.
>>>        while True:
>>>            value = func((yield), value)        # or here.
>>>    except YieldError:
>>
>> IIUC this works today if you substitute GeneratorExit and use
>> c.close() instead of next(c) below. (I don't recall why I split it out
>> into two different try/except blocks but it doesn't seem necessary.
>
> I tried it, c.close() doesn't work yet, but it does work with
> c.throw(GeneratorExit) :-)   But that still uses yield to get the value.

Yeah, sorry, I didn't mean to say that g.close() would return the
value, but that you can use GeneratorExit here. g.close() *does* throw
GeneratorExit (that's PEP 342); but it doesn't return the value yet. I
like adding that to PEP 380 though.

> I used a different way of starting the generator that checks for a value
> being yielded.
>
>
>
> class GeneratorStartError(TypeError): pass
>
> def start(g):
>    value = next(g)
>    if value is not None:
>        raise GeneratorStartError('started generator yielded a value')
>    return g

Whatever tickles your fancy. I just don't think this deserves a builtin.

> def reduce_collector(func):
>    value = None
>    try:
>        value = yield
>        while True:
>            value = func((yield), value)
>    except GeneratorExit:
>        yield value

Even today, I would much prefer using raise StopIteration(value) over
yield value (or yield Return(value)). Reusing yield to return a value
just looks wrong to me, there are too many ways to get confused (and
this area doesn't need more of that :-).

> def parallel_reduce(iterable, funcs):
>    collectors = [start(reduce_collector(func)) for func in funcs]
>    for v in iterable:
>        for coll in collectors:
>            coll.send(v)
>    return [c.throw(GeneratorExit) for c in collectors]
>
> def main():
>    it = range(100)
>    print(parallel_reduce(it, [min, max]))
>
> if __name__ == '__main__':
>    main()
>
>
>
>> As for being able to distinguish next(c) from c.send(None), that's a
>> few language revisions too late. Perhaps more to the point, I don't
>> like that idea; it breaks the general treatment of things that return
>> None and throwing away values. (Long, long, long ago there were
>> situations where Python balked when you threw away a non-None value.
>> The feature was boohed off the island and it's better this way.)
>
> I'm not sure I follow the relationship you suggest.  No values would be
> thrown away.  Or did you mean that it should be ok to throw away values? I
> don't think it would prevent that either.

Well maybe I was misunderstanding your proposed YieldError. You didn't
really explain it -- you just used it and assumed everybody understood
what you meant. My assumption was that you meant for YieldError to be
raised if yield was used as an expression (not a statement) but next()
was called instead of send().

My response was that it's ugly to make a distinction between

  x = <expr>
  del x  # Or just not use x

and

  <expr>

But maybe I misunderstood what you meant.

> What the YieldError case really does is give the generator a bit more
> control.  As far as the calling routine that uses it is concerned, it just
> works.  What happend inside the generator is completely transparent to the
> routine using the generator.  If the calling routine does see a YieldError,
> it means it probably was a bug.

That sounds pretty close to the rules for GeneratorExit.

>>>        # next was called not send.
>>>        yield value
>>
>> I object to overloading yield for both a *resumable* operation and
>> returning a (final) value; that's why PEP 380 will let you write
>> "return value". (Many alternatives were considered but we always come
>> back to the simple "return value".)
>
> That works for me.  I think lot of people will find it easy to learn.
>
>
>>> def parallel_reduce(iterable, funcs):
>>>    collectors = [reduce_collector(func) for func in funcs]
>>>    for v in iterable:
>>>        for coll in collectors:
>>>            coll.send(v)
>>>    return [next(c) for c in collectors]
>>
>> I really object to using next() for both getting the return value and
>> the next yielded value. Jacob's proposal to spell this as c.close()
>> sounds much better to me.
>
> If c.close also throws the GeneratorExit and returns a value, that would be
> cool. Thanks.

It does throw GeneratorExit (that's the whole reason for
GeneratorExit's existence :-).

> I take it that the objections have more to do with style and coding
> practices rather than what is possible.

Yeah, it's my gut and that's hard to reason with but usually right.
(See also: http://www.amazon.com/How-We-Decide-Jonah-Lehrer/dp/0618620117
)

>>> It nicely separates input and output parts of a co-function, which can be
>>> tricky to get right when you have to receive and send at the same yield.
>>
>> I don't think there was a problem with this in my code (or if there
>> was you didn't solve it).
>
> There wasn't in this code.  This is one of those areas where it can be
> really difficult to find the correct way to express a co-function that does
> both input and output, but not necessarily in a fixed order.

Maybe for that one should use a "channel" abstraction, like Go (and
before it, CSP)? I noticed that Monocle
(http://github.com/saucelabs/monocle) has a demo of that in its
"experimental" module (but the example is kind of silly).

> I begin almost any co-function with this at the top of the loop and later
> trim it up if parts of it aren't needed.
>
>   out_value = None
>   while True:
>      in_value = yield out_value
>      out_value = None
>      ...
>      # rest of loop to check in_value and modify out_value
>
> As long as None isn't a valid data item, this works most of the time.
>
>
>>> Maybe in Python 4k?  Oh well. :-)
>>
>> Nah.
>
> I'm ok with that.
>
> Ron
>



-- 
--Guido van Rossum (python.org/~guido)