looping over a big file

Jp Calderone exarkun at divmod.com
Mon Jul 4 00:06:25 CEST 2005

On Sun, 3 Jul 2005 23:52:12 +0200, martian <nospam at hetnet.nl> wrote:
>I've a couple of questions regarding the processing of a big text file
>1) how does python handle:
>> for line in big_file:
>is big_file all read into memory or one line is read at a time or a buffer
>is used or ...?

It uses an internal buffer to reach a happy medium between performance and memory usage.

>2) is it possible to advance lines within the loop? The following doesn't
>> for line in big_file:
>         line_after  = big_file.readline()

Yes, but you need to do it like this:

  fileIter = iter(big_file)
  for line in fileIter:
      line_after = fileIter.next()

  Don't mix iterating with any other file methods, since it will confuse the buffering scheme.


More information about the Python-list mailing list