Database in a simple ascii file

Cameron Laird claird at
Wed Nov 28 18:04:46 CET 2001

In article <3C04F826.CB408C9D at>,
Peter Hansen  <peter at> wrote:
>Rafael Mentz Aquino wrote:
>> How many lines can I have in a simple ascii file before
>> I start to loose performance when I open this file and
>> put all the lines in a dict, turning the first column into the key
>> and the rest of the line into a list associated to that key.
>> (my file has just 100 lines, but that question starts to bother me ;-)
>You will start to lose performance, relatively to no lines,
>with even one line.  If you start with ten lines, you will
>begin losing performance after you switch to a file of 11 
>Why focus on performance when you have no idea how fast
>you want your program to be?  Recent advances in software
>development are teaching programmers *never* to optimize
>until the program works, and only when there is a clear
>requirement for more performance.
>If you are really concerned about performance, you will 
>always be able to get more than a 10 times speedup by 
>switching to a compiled language over Python.  Use of
>Python should imply you are more interested in readability
>and ease of use, and thus speed of *development*, over
>speed of execution.
>(Shorter answer: don't grow any grey hairs over it.)
Peter and I agree on so many large conclusions,
and differ on so many details.

My own suspicion about the situation the original
poster describes is that I/O time dominates, and
that neither "switching to a compiled language" nor
a fancier persistence method will help by even a
factor of two (maybe not 10%) in comparison with
judiciously straightforward Python coding.  I know
that's true for many things I do:  it just takes a
long time to read in data.  Once it's in memory,
Python manages it plenty swiftly.

Peter's entirely correct in his general advice.

Cameron Laird <Cameron at>

More information about the Python-list mailing list