Can you do it faster? (parsing text file)

[Andrew Bennetts]

On Tue, Jan 28, 2003 at 11:16:04AM +0000, Marcus Stojek wrote:
> Hi,
> 
> I have to parse large txt-files (15Mb and more, 200000 lines).
> The file consits of comma separated keywords and alphanumerical data.

While I see you have some quirky requirements, I'd recommend:
  - Use the CSV to parse the fields:
    http://www.object-craft.com.au/projects/csv/
  - Read the file with "for line in myFile:"  (you're using Python 2.2, I
    presume?)

I suspect with those you could get a nice speed boost.  You could also try
mucking around with Psyco and/or the Python 2.3 alpha (Python 2.3 is faster
than older versions).

There's various other micro-optimisations that could be made, but they're
probably not worth it.

-Andrew.