generator expressions: performance anomaly?

John Machin sjmachin at lexicon.net
Sun Jan 16 00:46:21 EST 2005


Please consider the timings below, where a generator expression starts
out slower than the equivalent list comprehension, and gets worse:

>python -m timeit -s "orig=range(100000)" "lst=orig[:];lst[:]=(x for x
in orig)"
10 loops, best of 3: 6.84e+004 usec per loop

>python -m timeit -s "orig=range(200000)" "lst=orig[:];lst[:]=(x for x
in orig)"
10 loops, best of 3: 5.22e+005 usec per loop

>python -m timeit -s "orig=range(300000)" "lst=orig[:];lst[:]=(x for x
in orig)"
10 loops, best of 3: 1.32e+006 usec per loop

>python -m timeit -s "orig=range(100000)" "lst=orig[:];lst[:]=[x for x
in orig]"
10 loops, best of 3: 6.15e+004 usec per loop

>python -m timeit -s "orig=range(200000)" "lst=orig[:];lst[:]=[x for x
in orig]"
10 loops, best of 3: 1.43e+005 usec per loop

>python -m timeit -s "orig=range(300000)" "lst=orig[:];lst[:]=[x for x
in orig]"
10 loops, best of 3: 2.33e+005 usec per loop

Specs: Python 2.4, Windows 2000, 1.4GHz Athlon chip, 768Mb of memory.

Background: There was/is a very recent thread about ways of removing
all instances of x from a list. /F proposed a list comprehension to
build the result list. Given a requirement to mutate the original list,
this necessitates the assignment to lst[:]. I tried a generator
expression as well. However while the listcomp stayed competitive up to
a million-element list, the genexp went into outer space, taking about
20 times as long. The above timeit runs show a simpler scenario where
the genexp also seems to be going quadratic.
Comments, clues, ... please.

TIA,
John




More information about the Python-list mailing list