yoav glazner wrote:

On Fri, Jun 19, 2009 at 6:11 AM, Steven D'Aprano mailto:steve@pearwood.info> wrote:

On Fri, 19 Jun 2009 07:02:12 am Terry Reedy wrote:

As I see it, the correct solution for "my function needs to iterate over an iterable twice" is not to expect the caller to pass a sequence, but to convert the iterable to a sequence inside your function:

def function(iterable): # Iterate over iterable twice L = list(iterable) for _ in (1, 2): for x in L: pass

As I see it the correct solution is def function(iterable): # Iterate over iterable twice L = iterable.clone() #can be done?

For non-iterator iterables, it is not necessary. For iterators in general, it is difficult at best. For iterators returned by a constructor, it should be much easier to re-call the constructor.

2009/6/19 John Graham :

Exactly how does itertools.tee not cover this use case?

I've been wondering the same sort of thing. But if you're going to sequentially run through the iterator twice for i in it: do something for i in it: do something else then itertools.tee is documented as being basically equivalent to, but worse than, list(): "In general, if one iterator is going to use most or all of the data before the other iterator, it is faster to use list() instead of tee()". So the question becomes, how does list() not cover this use case. The only answer I can see is that the proposal avoids allocating temporary storage for all of the values generated. Consider xrange(1000000) - if you can call xrange twice, rather than saving a million values, you've saved a lot of memory. But this only applies when you're writing functions designed to take extremely general iterators. Someone writing a function which can take essentially arbitrary iterators, and which has to handle unboundedly large input gracefully, probably has worse problems than the lack of standard library support for something they can code themselves fairly easily... Paul.

Mathias Panzenböck wrote:

Paul Moore wrote:

...

2009/6/19 John Graham :

...

Exactly how does itertools.tee not cover this use case?

I've been wondering the same sort of thing. But if you're going to sequentially run through the iterator twice

for i in it: do something for i in it: do something else

then itertools.tee is documented as being basically equivalent to, but worse than, list(): "In general, if one iterator is going to use most or all of the data before the other iterator, it is faster to use list() instead of tee()".

So the question becomes, how does list() not cover this use case.

The only answer I can see is that the proposal avoids allocating temporary storage for all of the values generated. Consider xrange(1000000) - if you can call xrange twice, rather than saving a million values, you've saved a lot of memory.

But this only applies when you're writing functions designed to take extremely general iterators. Someone writing a function which can take essentially arbitrary iterators, and which has to handle unboundedly large input gracefully, probably has worse problems than the lack of standard library support for something they can code themselves fairly easily...

Paul.

I don't know if this applies here but generators can also yield infinite lists. In such an case list() is not an alternative.

-panzi

Hrm, that leads to the question, is it mathematically possible to iterate over an infinite list... twice? :) -John

John Graham writes:

Hrm, that leads to the question, is it mathematically possible to iterate over an infinite list... twice? :)

Of course. Some infinite ordinals are bigger than others, you know. :)

Greg Ewing wrote:

John Graham wrote:

...

Hrm, that leads to the question, is it mathematically possible to iterate over an infinite list... twice? :)

Sure, as long as you don't insist on finishing the first iteration before starting the second one...

And it takes exactly the same amount of time as doing it once. :-)

Thats what I was thinking about. Like: def foo(seq): for x, y, z in izip(seq, islice(seq,1,None,2), islice(seq,2,None,3)): ...

Lie Ryan wrote:

Steven D'Aprano wrote:

...

On Sat, 20 Jun 2009 01:55:42 pm John Graham wrote:

...

Hrm, that leads to the question, is it mathematically possible to iterate over an infinite list... twice? :) Mathematically, yes. Practically, no.

Mathematically, the trick is to ensure that the time it takes to iterate over each item approaches zero sufficiently fast. E.g. if it takes 1 unit of time to process the first item, and 1/2 units of time for the second, and 1/4 for the third, 1/8 for the fourth, etc, then you can iterate over an infinite number of items in 2 units of time total. Then you can do them all again :)

It is also possible to do parallel iteration, but I'm not sure that mathematically parallel processing makes sense. I'm not even sure that mathematically there is such thing as iteration (that does not involve recursion).

Well there is something called "process algebra" (at least I think its called that way, in german its: Prozess Algebra). This provides a (mathematical?) model for such things.