j: Next unread message
k: Previous unread message
j a: Jump to all threads
j l: Jump to MailingList overview
[Donald 'Paddy' McCarthy]
Hi, I have a blog entry, http://paddy3118.blogspot.com/2009/05/pipe-fitting-with-python-generators.ht..., in which I define a helper function to get around the problem of reversion in the nesting of generators and the proliferation of nested brackets.
See the blog entry for a fuller treatment, but in essence if you have a series of generators, and want the data to conceptually flow like this:
gen1 -> gen2 -> gen3 -> gen4 ...
You have to write:
With pipe, you would write:
pipe(gen1, gen2, gen3, gen4, ...)
I'm not too excited about this for several reasons.
Many itertools do not stack linearly. The groupby() tool emits a stream of (key, generator) pairs that do not readily feed into other itertools (the generator part is meant to be consumed right away). The tee() tool emits multiple streams meant to be consumed contemporaneously. The zip tools take in multiple input steams. Some like count() and repeat() are meant to be feed into zip alongside other streams. IOW, piping is only a natural model for a limited subset of use cases.
The existing approaches let you handle multiply nested tools by simply assigning intermedate results to variables:
it1, it2 = tee(iterable) decorated = izip(imap(func, it1), count(), it2) processed = sometool(decorated, somearg) undecorated = imap(itemgetter(2), processed) return undecorated
I don't like conflating the mental models for pipes with those for generators. While there are similarities, there are also differences that become less evident when a piping notation is used. Operating system pipes are buffered and producer processes can get suspended while consumers catch up. With generators, the consumer functions are in-charge, not the producers.
There doesn't seem to be much of an advantage to a pipe notation for generators. No new capabilites are added. It is essentially just a new syntax for a subset of things we already do. And, the notation becomes awkward and forced for use cases with multiple input streams and multiple output streams. Why add yet another way to do it?