huge file -> memory problem

Fred Gansevles gansevle at cs.utwente.nl
Mon Mar 13 07:59:21 EST 2000


Jørn Eilertsen wrote:

> Hi,
>
> I have this program that's supposed to wade through a rather large file
> (700MB and growing) line by line searching for a pattern. The problem
> is, when i open the file to read it using f = open(file, 'r'), it just
> takes all available memory which obviously isn't acceptable. And now the
> question; is there a way to open and read a file which isn't doesn't
> require that much memory. It seems by doing it the way I am the whole
> file is read into memory.
>
> Any help would be greatly apreciated.
>
> Jørn

In the case of a BIG file, reading the whole file consumes all the
available memory.
Since you search the file on a line-by-line basis, using 'readline()' is
the simplest solution.

Your loop looks like:

        fp = open (file, 'r')
        while 1:
                line = fp.readline ()
                if not line:    # EOF
                        break
                # if a trailing '\n' is a problem; chop it here
                if line[-1:] == '\n': line = line[:-1]
                process (line)






More information about the Python-list mailing list