[Python-Dev] iterzip()

Raymond Hettinger python@rcn.com
Sun, 28 Apr 2002 23:04:04 -0400


From: "Guido van Rossum" <guido@python.org>
> > I'm already working on a separate module for iterators galore
> > (and will cross-check to Haskell to make sure I didn't miss anything).
> 
> +!
> 
> > I posted this one separately because zip() eats memory like crazy
> > and because a Python generator version crawls like a snail.
> 
> Do you have use cases where the memory use matters?  I.e. where it
> needs more memory than you have RAM?

No.  Am I'm not proposing a backport to Pippy ;)

> 
> > IMHO, This is a better way to loop over multiple sequences and
> > has a chance at becoming the tool of choice.  I scanned all of my
> > Python code and found that iterzip() was a better choice in every
> > case except a matrix transpose coded as zip(*mat).
> 
> Did you time any of these?

I just timed it and am shocked.  iterzip() has exactly the same code
as zip() except for the final append result to list.  So, I expected only
a microscopic speed-up.  I don't see where the huge performance
improvement came from.

First timing:
----------------------
19.439999938 zip
1.43000006676 iterzip
30.1000000238 genzip

Second timing:
----------------------
20.0999999046 zip
1.60000002384 iterzip
29.0599999428 genzip

Timing code:
---------------
# time iterzip
from __future__ import generators
import time

def run(f, iterables):
    start = time.time()
    for tup in f(*iterables):
        pass
    return time.time() - start

def genzip(*iterables):
    iterables = map(iter, iterables)
    while 1: 
        yield tuple([i.next() for i in iterables])  

n = 1000000
for f in [zip, iterzip, genzip]:
    print run(f, [xrange(n), xrange(n), xrange(n)]), f.__name__



Raymond Hettinger