python reading file memory cost

Tony Zhang warriorlance at
Mon Aug 1 23:22:50 EDT 2011


Actually, I used .readline() to parse file line by line, because I need
to find out the start position to extract data into list, and the end
point to pause extracting, then repeat until the end of file.
My file to read is formatted like this:


data block(e.g. 10 cols x 1000 rows)
blank line
data block(e.g. 10 cols x 1000 rows)
blank line
let's call this file 'myfile'
and my python snippet:

blocknum=0 #number the data block
while True"
	# find the extract begnning
	while not f.readline().startswith('/a1/'):pass
	# creat multidimensional list to store data block	
	blocknum +=1

	while line.strip():  
	# check if the line is a blank line, i.e the end of one block
		data[blocknum-1].append(["2.6E" %float(x) for x in line.split()])
		line = f.readline()
	print "Read Block %d" %blocknum
	if not f.readline(): break 

The running result was that read a 500M file consume almost 2GB RAM, I
cannot figure it out, somebody help!
Thanks very much!


More information about the Python-list mailing list