Fast File Input

[P at draigBrady.com]

Scott Brady Drummonds wrote:
> Hi, everyone,
> 
> I'm a relative novice to Python and am trying to reduce the processing time
> for a very large text file that I am reading into my Python script.  I'm
> currently reading each line one at a time (readline()), stripping the
> leading and trailing whitespace (strip()) and splitting it's delimited data
> (split()).  For my large input files, this text processing is taking many
> hours.
> 
> If I were working in C, I'd consider using a lower level I/O library,
> minimizing text processing, and reducing memory redundancy.  However, I have
> no idea at all what to do to optimize this process in Python.
> 
> Can anyone offer some suggestions?

This actually improved a lot with python version 2
but is still quite slow as you can see here:
http://www.pixelbeat.org/readline/
There are a few notes within the python script there.

Pádraig.