On 10/25/2010 10:13 AM, Guido van Rossum wrote:
[Changed subject]
On 2010-10-25 04:37, Guido van Rossum wrote:
This should not require threads.
Here's a bare-bones sketch using generators: [...]
On Mon, Oct 25, 2010 at 3:19 AM, Jacob Holm
wrote: If you don't care about allowing the funcs to raise StopIteration, this can actually be simplified to: [...]
Indeed, I realized this after posting. :-) I had several other ideas for improvements, e.g. being able to pass an initial value to the reduce-like function or even being able to supply a reduce-like function of one's own.
More interesting (to me at least) is that this is an excellent example of why I would like to see a version of PEP380 where "close" on a generator can return a value (AFAICT the version of PEP380 on http://www.python.org/dev/peps/pep-0380 is not up-to-date and does not mention this possibility, or even link to the heated discussion we had on python-ideas around march/april 2009).
Can you dig up the link here?
I recall that discussion but I don't recall a clear conclusion coming from it -- just heated debate.
Based on my example I have to agree that returning a value from close() would be nice. There is a little detail, how multiple arguments to StopIteration should be interpreted, but that's not so important if it's being raised by a return statement.
Assuming that "close" on a reduce_collector generator instance returns the value of the StopIteration raised by the "return" statements, we can simplify the code even further:
def reduce_collector(func): try: outcome = yield except GeneratorExit: return None while True: try: val = yield except GeneratorExit: return outcome outcome = func(outcome, val)
def parallel_reduce(iterable, funcs): collectors = [reduce_collector(func) for func in funcs] for coll in collectors: next(coll) for val in iterable: for coll in collectors: coll.send(val) return [coll.close() for coll in collectors]
Yes, this is only saving a few lines, but I find it *much* more readable...
I totally agree that not having to call throw() and catch whatever it bounces back is much nicer. (Now I wish there was a way to avoid the "try..except GeneratorExit" construct in the generator, but I think I should stop while I'm ahead. :-)
This is how my mind wants to write this. @consumer def reduce_collector(func): try: value = yield # No value to yield here. while True: value = func((yield), value) # or here. except YieldError: # next was called not send. yield = value def parallel_reduce(iterable, funcs): collectors = [reduce_collector(func) for func in funcs] for v in iterable: for coll in collectors: coll.send(v) return [next(c) for c in collectors] It nicely separates input and output parts of a co-function, which can be tricky to get right when you have to receive and send at the same yield. Maybe in Python 4k? Oh well. :-)
The interesting thing is that I've been dealing with generators used as coroutines or tasks intensely on and off since July, and I haven't had a single need for any of the three patterns that this example happened to demonstrate:
- the need to "prime" the generator in a separate step
Having a consumer decorator would be good. def consumer(f): @wraps(f) def wrapper(*args, **kwds): coroutine = f(*args, **kwds) next(coroutine) return coroutine return wrapper Or maybe it would be possible for python to autostart a generator if it's sent a value before it's started? Currently you get an almost useless TypeError. The reason it's almost useless is unless you are testing for it right after you create the generator, you can't (easily) be sure it's not from someplace inside the generator. Ron