[Python-ideas] Include partitioning in itertools

[Chris Angelico]

On Mon, Nov 2, 2015 at 2:42 PM, David Mertz <mertz at gnosis.cx> wrote:
> I'm surprised to have just noticed that `itertools` is oddly broken this way
> (under Python 3.4.3):
>
> This is perfectly happy:
>
>>>> def fibs():
>
>     a, b = 1, 1
>     while True:
>         yield b
>         b, a = a+b, b
> ...
>>>> f = ((a,b) for a in fibs() for b in fibs())
>>>> next(f)
> Out[3]: (1, 1)
>>>> next(f)
> Out[4]: (1, 2)
>>>> next(f)
> Out[5]: (1, 3)
>
> And yet this freezes be greedily trying to go through the infinite iterator:
>
>>>> from itertools import product
>>>> product(fibs(), fibs())

Yeah. The problem is that you're looking at a simplified version of
the reference implementation. Further down it says that it's
equivalent to a rather longer function, which (critically) includes a
call to tuple(), to coalesce the iterables into their values. When you
use the naive genexp, what you're actually doing is constructing a new
instance of b's fibs() for every iteration of a's one (you won't see
that on an infinite iterator, but try a finite iterator with some
print() calls and you'll see); itertools.product obviously can't do
that, as it's given the iterable and not the means of constructing
more. The tiny example up the top is inaccurate in its handling of
edge cases, but for stable, repeatable iterables, it will return the
same results.

ChrisA