[Tutor] list comprehension efficiency

[Steven D'Aprano]

Bill Allen wrote:
> Generally speaking, are list comprehensions more efficient that the
> equivalent for loop with interior conditionals for a given task?  Do they
> compile down to the same Python byte code?

It depends, and no.

For-loops are more general, so there are loops that can't be written as list 
comprehensions. Consequently, so is the byte code generated. Python makes no 
attempt to analyse a for-loop and decide that it could be written as a list 
comp. Nor do list comps generate the exact same code as a for-loop: they 
can't, because the semantics are slightly different.

(Specifically, a list comp is an atomic operation: either the whole thing 
succeeds, or the resultant list does not persist. That same does not apply to 
for-loops.)

You can inspect the byte code produced by using the dis module.

But, for the subset of possible for-loops which can be re-written as list 
comps, yes, list comps often are more efficient, as building the list can be 
performed at full C speed instead of at relatively slower Python speed. But 
don't over-estimate how big a difference that makes in practice. For trivial 
loops, it makes a big difference:

py> with Timer():
...     _ = [None for i in range(10000)]
...
time taken: 0.004692 seconds
py>
py> with Timer():
...     _ = []
...     for i in range(10000):
...             _.append(None)
...
time taken: 0.020877 seconds


but for loops that do actual, significant work, the difference becomes 
correspondingly less important:

py> def func(x):
...     y = 2*i**3 - 15*i*i + 17*i + 3
...     z = x**0.5/(2*y**2 - 1)
...     return (3*z)/(1+z)**1.25
...
py> with Timer():
...     _ = [func(i) for i in range(10000)]
...
time taken: 0.210438 seconds
py>
py> with Timer():
...     _ = []
...     for i in range(10000):
...             _.append(func(i))
...
time taken: 0.250344 seconds


If you're interested in using my Timer() code, you can find it here:

http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-statement/


Beware though, that you can't exit out of a list comp early. So if you have 
code like this:


collection = []
for x in sorted(sequence):
     if x > 10:
         break
     collection.append(x + 5)


versus this:


collection = [x+5 for x in sorted(sequence) if x <= 10]

Consider what happens if sequence = range(1000000). The for-loop does eleven 
iterations, the list-comp does a million. Guess which will be faster? :)


-- 
Steven