Memory error due to big input file
davea at ieee.org
Mon Jul 13 23:01:17 CEST 2009
sityee kong wrote:
> Hi All,
> I have a similar problem that many new python users might encounter. I would
> really appreciate if you could help me fix the error.
> I have a big text file with size more than 2GB. It turned out memory error
> when reading in this file. Here is my python script, the error occurred at
> line -- self.fh.readlines().
> import math
> import time
> class textfile:
> def __init__(self,fname):
> def myfun(snp,start,end):
> for row in subdata:
> if (k=="0/0") & (k!="0/0"):
> elif (k!="0/0") & (k=="0/0"):
> elif (k!="0/0") & (k!="0/0"):
> result.write(snp+" "+str(NEWmiss)+" "+str(OLDmiss)+" "+str(DIFF)+"\n")
> result.write("SNP NEWmiss OLDmiss DIFF\n")
> for i in range(lfile):
> if (i==0): continue
> if (before==after):
> if (i!=(lfile-1)): continue
> if (i ==(lfile-1)):
> sincerely, phoebe
Others have pointed out that you have too little memory for a 2gig data
structure. If you're running on a 32bit system, chances are it won't
matter how much memory you add, a process is limited to 4gb, and the OS
typically takes about half of it, your code and other data takes some,
and you don't have 2gig left. A 64 bit version of Python, running on a
64bit OS, might be able to "just work."
Anyway, loading the whole file into a list is seldom the best answer,
except for files under a meg or so. It's usually better to process the
file in sequence. It looks like you're also making slices of that data,
so they could potentially be pretty big as well.
If you can be assured that you only need the current line and the
previous two (for example), then you can use a list of just those three,
and delete the oldest one, and add a new one to that list each time
through the loop.
Or, you can add some methods to that 'textfile' class that fetch a line
by index. Brute force, you could pre-scan the file, and record all the
file offsets for the lines you find, rather than storing the actual
line. So you still have just as big a list, but it's a list of
integers. Then when somebody calls your method, he passes an integer,
and you return the particular line. A little caching for performance,
and you're good to go.
Anyway, if you organize it that way, you can code the rest of the module
to not care whether the whole file is really in memory or not.
BTW, you should derive all your classes from something. If nothing
else, use object.
More information about the Python-list