Python 'for' loop is memory inefficient

Steven D'Aprano steve at REMOVE-THIS-cybersource.com.au
Sat Aug 15 14:25:00 CEST 2009


On Fri, 14 Aug 2009 18:25:45 -0700, Dr. Phillip M. Feldman wrote:

> It seems as though Python is actually expanding range(2,n) into a list
> of numbers, even though this is incredibly wasteful of memory. There
> should be a looping mechanism that generates the index variable values
> incrementally as they are needed.


Others have already pointed out to you that xrange() (for Python 2.x) 
does what you want. In Python 3.x, the old range() is gone for good, and 
xrange() renamed to just range().

However, I'd like to point out that your subject line is fundamentally 
incorrect. What you've discovered has *nothing* to do with for-loops: the 
for-loop will happily iterate over whatever object you pass to it (or at 
least try to, because not all objects are iterable). You might be 
thinking that Python requires you to write for-loops as 

for i in range(...):

but that's not correct. The for-loop doesn't care what object comes after 
the "in" and before the ":", so long as it can iterate over it one item 
at a time:

for i in [1, 3, 4, 5, 2, 0, -1, 8, 7, 6]:
    print i

for c in "abcd":
    print c

and many other variations will work perfectly fine.


The memory-consumption you have discovered is a property of the range() 
function, not the for-loop:

>>> L = range(2, 20)
>>> L
[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

Compare to xrange:

>>> L = xrange(2, 20)
>>> L
xrange(2, 20)
>>> list(L)  # expand out into a list
[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]


By the way, both range() and xrange() take an optional 'stepsize' 
argument, defaulting to 1:

>>> range(3, 20, 2)
[3, 5, 7, 9, 11, 13, 15, 17, 19]


help(range) and help(xrange) in the interactive interpreter are your 
friends.



-- 
Steven



More information about the Python-list mailing list