memory problem with list creation
allardwarrink at gmail.com
Wed Jan 13 15:24:04 CET 2010
Within a python script I'm using a couple of different lists
containing a large number of floats (+8M). The execution of this
script fails because of an memory error (insufficient memory).
I thought this was strange because I delete all lists that are not
longer necessary directly and my workstation theoretically has more
than enough memory to run the script.
so I did some investigation on the memory use of the script. I found
out that when i populated the lists with floats using a for ... in
range() loop a lot of overhead memory is used and that this memory is
not freed after populating the list and is also not freed after
deleting the list.
This way the memory keeps filling up after each newly populated list
until the script crashes.
I did a couple of tests and found that populating lists with range or
xrange is responsible for the memory overhead.
Does anybody know why this happens and if there's a way to avoid this
First the line(s) python code I executed.
Then the memory usage of the process:
Mem usage after creation/populating of big_list
Mem usage after deletion of big_list
big_list = [0.0] * 2700*3250
big_list = [0.0 for i in xrange(2700*3250)]
big_list = [0.0 for i in range(2700*3250)]
big_list = [float(i) for i in xrange(2700*3250)]
big_list = [float(i) for i in range(2700*3250)]
big_list = [i for i in xrange(2700*3250)]
big_list = [i for i in range(2700*3250)]
big_list = 
for i in range(2700*3250):
big_list = 
for i in xrange(2700*3250):
More information about the Python-list