[Python-ideas] pipe function for itertools?

[Steven D'Aprano]

On Tue, 26 May 2009 08:32:58 pm Donald 'Paddy' McCarthy wrote:
> Hi, I have a blog entry,
> http://paddy3118.blogspot.com/2009/05/pipe-fitting-with-python-genera
>tors.html, in which I define a helper function to get around the
> problem of reversion in the nesting of generators and the
> proliferation of nested brackets.
>
> See the blog entry for a fuller treatment, but in essence if you have
> a series of generators, and want the data to conceptually flow like
> this:
>
> gen1 -> gen2 -> gen3 -> gen4 ...
>
> You have to write:
>
> ...(gen4(gen3(gen2(gen1()))))...
>
> With pipe, you would write:
>
> pipe(gen1, gen2, gen3, gen4, ...)
> 
> Which you could use like this:
>
> for data in pipe(...): do_something_with_data()
>
>
> If I use dots for indentation, then maybe the definition will come
> through to the group:
>
> def pipe(*cmds):
> ....gen = cmds[0]
> ....for cmd in cmds[1:]:
> ....... gen = cmd(gen)
> ....for x in gen:
> ....... yield x


The function signature is misleading. It doesn't take a series of 
generator functions ("cmds"), it takes an initial iterable followed by 
a series of generator functions.

It seems to me that a cleaner definition would be:

def pipe(iterable, *generators):
    for gen in generators:
        iterable = gen(iterable)
    for x in iterable:
        yield x


This does seem to be a special case of function composition. If 
functools grew a compose() function, you could write:

from functools import compose
def pipe(it, *gens):
    for x in compose(*gens)(it):
        yield x

Here's an untested definition for compose:

def compose(f, *funcs, **kwargs):
    if not funcs:
        raise TypeError('compose() requires at least two functions')
    if kwargs.keys() not in ([], ['doc']):
        # I wish Python 2.5 had keyword only args...
        raise TypeError('bad keyword argument(s)')
    def function_composition(*args, **kwargs):
        value = f(*args, **kwargs)
        for g in funcs:
            value = g(value)
        return value
    function_composition.__doc__ = kwargs.get('doc')
    return function_composition

which is more complicated than your version, of course, but also more 
general.



> A couple of readers thought that it might be a good tool for
> itertools to have.

Do you have any other use-cases?




-- 
Steven D'Aprano