[issue10109] itertools.product with infinite iterator cause MemoryError.

[Lucas Wiman]

Lucas Wiman added the comment:

I realize this is an old (and closed!) thread, but here are a few notes from running into this issue recently, working on a search problem with infinite iterators.

First, note that yegle's recursive solution is not correct, since it exhausts the iterators:
>>> def product(*args):
...     if len(args) == 1:
...         for i in args[0]:
...             yield [i]
...     else:
...         for i in args[0]:
...             for j in product(*args[1:]):
...                 j.append(i)
...                 yield j
... 
>>> 
>>> assert len(list(itertools.product(*[iter(range(2)) for _ in range(10)]))) == 2 ** 10
>>> assert len(list(product(*[iter(range(2)) for _ in range(10)]))) == 2 ** 10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

This is fairly tricky to get right, and even a correct fully-lazy solution requires each iterator to eventually be fully stored in memory. For that reason, Sumudu's MemoryError is not something that can be easily escaped, though it can be delayed.

For infinite iterators, the naive "lexicographic" solution to product(count(0), count(0)) would never get to 1 on the first iterator, yielding (0, 0), (0, 1), (0, 2), ...  To fully explore the space, the user needs to think about how to iterate over that space, so IMO keeping infinite iterators as invalid arguments to itertools.product makes sense.

----------
nosy: +Lucas Wiman

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10109>
_______________________________________