Cropping log files

Robin Munn rmunn at
Tue Dec 16 21:57:41 CET 2003

Kamus of Kadizhar <yan at> wrote:
> Yet another possible newbie question.
> I'm tyring to figure out how to best crop a log file.  I have a file 
> that grows infinitely in length.  I want to keep only the last n entries 
> in the file.
> I don't want to suck the whole file into memory if I can help it, and I 
> can't help thinking that doing a
> for line in file(logfile)
>     nr += 1
> to count the number of lines, then reading the file again, discarding 
> the first nr - n records, writing the rest to a temp file, and then 
> renaming the files is the most efficient way to go.

Are you in a Unix environment here? If so, why not let the 'tail'
command do the work for you? Something like this:

    # Close any open file objects pointing to the log file
    long_log_filename = '/var/log/lotsoflines.log'
    short_log_filename = '/var/log/lotsoflines.log.truncated'
    lines_to_keep = 250
    cmd = 'tail -%(lines)d %(long)s > %(short)s && mv %(short)s %(long)s' % {
                  'lines': lines_to_keep,
                  'short': short_log_filename,
                  'long':  long_log_filename,
    # Now reopen your file objects

Or if you really want to do it in pure Python, then have a look at the
source for 'tail' and see how it finds the last N lines. In most of my
experiments, I've found that 'tail' grabs the last 10 lines out of a
1-gigabyte log file within milliseconds, while 'wc -l' takes a lot
longer to count the whole file. That's not very scientific evidence, of
course, but why don't you try it and see for yourself?

Robin Munn
rmunn at

More information about the Python-list mailing list