<br>You could try forcing a garbage collection...<br><br><div class="gmail_quote">On Mon, Aug 1, 2011 at 8:22 PM, Tony Zhang <span dir="ltr"><<a href="mailto:warriorlance@gmail.com">warriorlance@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Thanks!<br>
<br>
Actually, I used .readline() to parse file line by line, because I need<br>
to find out the start position to extract data into list, and the end<br>
point to pause extracting, then repeat until the end of file.<br>
My file to read is formatted like this:<br>
<br>
blabla...useless....<br>
useless...<br>
<br>
/sign/<br>
data block(e.g. 10 cols x 1000 rows)<br>
...<br>
blank line<br>
/sign/<br>
data block(e.g. 10 cols x 1000 rows)<br>
...<br>
blank line<br>
...<br>
...<br>
EOF<br>
let's call this file 'myfile'<br>
and my python snippet:<br>
<br>
f=open('myfile','r')<br>
blocknum=0 #number the data block<br>
data=[]<br>
while True"<br>
# find the extract begnning<br>
while not f.readline().startswith('/a1/'):pass<br>
# creat multidimensional list to store data block<br>
data=append([])<br>
blocknum +=1<br>
line=f.readline()<br>
<br>
while line.strip():<br>
# check if the line is a blank line, i.e the end of one block<br>
data[blocknum-1].append(["2.6E" %float(x) for x in line.split()])<br>
line = f.readline()<br>
print "Read Block %d" %blocknum<br>
if not f.readline(): break<br>
<br>
The running result was that read a 500M file consume almost 2GB RAM, I<br>
cannot figure it out, somebody help!<br>
Thanks very much!<br>
<br>
--Tony<br>
<br>
</blockquote></div><br>