[Python-ideas] iterating over a generator while sending values

[Clay Gerrard]

Hi,

I often use generators when I want to bake some smarts into my producers in
a typical producer and consumer pattern.  Twice in the past year or so on
different projects after a few iterations on my producer/consumer
code ("iterations", ha!) I would eventually find I wanted my consumer to
send some simple feedback to the producer - when using generators send
works great for this!

I normally start here:

q = producer()

def consume(q):
    for item in q:
        success = handle(item)

But then ended up here:

def consume(q):
    success = None
    while True:
        try:
            item = q.send(success)
        except StopIteration:
            break
        success = handle(item)

But this always feels clunky; almost "out of order" - I've normally got
some logging and other bits in there.  I'm just trying to iterate over the
queue, the StopIteration just feels like noise.

It seems I always started out with a simple iteration and then end up
refactoring in the send function once I realize I need to add something
extra into my producer:

I want to just say:

def consume(q):
    for item in q:
        success = handle(item)
        q.send(success)  # oh how I wish I could just add this here!

But I can just "tack on" the send because it consumes too, so I have to
rephrase the "simple" iteration into a "while True try except StopIteration
break" - yuk.

I normally just end up adding a little helper function, which I've named
"sending" and use it like this:

def consume(q):
    for item in sending(q, lambda: success):
        success = handle(item)

^ much closer to the original iterative code!

Here's the implementation if you didn't already guess:

# this function is missing from stdlib for some reason...
def sending(g, sender):
    """
    Iterate over g with send

    :params g: the iterable
    :params sender: the callable returning the value to send
    """
    yield g.next()
    while True:
        yield g.send(sender())

This will raise StopIteration as soon as g is exhausted, but since the
caller is already consuming "sending" with a for loop it all works ok in
the end.

Does anyone else have a different way of iterating over a generator while
sending values?  Does everyone just already have a similar function stuck
in their bag?  Does this happen to other folks frequently enough that a
generalizable solution could be added to stdlib?  Maybe it's too simple to
be worth anyone's time or i'm missing something that already works better...

Thanks!

-clayg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140327/6748d86d/attachment.html>