Why is text file processing SO slow?

jaimewyant at my-deja.com jaimewyant at my-deja.com
Wed Sep 20 22:10:36 CEST 2000


I have a program that processes large files ~25 megs.  The algorithm
that I use is as follows:

while not EOF:
    readline
    store line in list

    if line == 'End of set' marker
        dump the list to appopriate file (produces 2 sep files)
        empty the list
    elsif line contains 'EDS' regexp
        set the eds flag


The above algorithm is only slightly stripped down.  The above program
written in Python takes 55 seconds to execute.  The same program
written in perl takes 15 seconds.  The C version takes 10 seconds.

Is it typically a BAD idea to use Python (1.5.2) to process text files?

TIA,
jaime


Sent via Deja.com http://www.deja.com/
Before you buy.



More information about the Python-list mailing list