Help with script with performance problems

Dennis Roberts googlegroups at spacerodent.org
Sun Nov 23 13:58:20 EST 2003


Ville Vainio <ville.spammehardvainio at spamtut.fi> wrote in message news:<du7r7zz4ei3.fsf at amadeus.cc.tut.fi>...
> > f = sys.stdin
> 
> Have you tried using a normal file instead of stdin? BTW, you can
> iterate over a file easily by "for line in open("mylog.log"):". ISTR
> it's also more efficient than readline()'s, because it caches the
> lines instead of reading them one by one. You can also get the line
> numbers by doing "for linenum, line in enumerate(open("mylog.log")):"
> 

i have a 240207 line sample log file that I test with.  The script I
submitted parsed it in 18 seconds.  My perl script parsed it in 4
seconds.

The new python script, using a normal file as suggested above, does it
in 3 seconds!

Changed "f = sys.stdin" to "f = open('sample', 'r')".

Thanks Ville!

Note (I made the other changes one at a time as well - the file open
change was the only one that made it faster)




More information about the Python-list mailing list