Database in a simple ascii file
claird at starbase.neosoft.com
Wed Nov 28 18:04:46 CET 2001
In article <3C04F826.CB408C9D at engcorp.com>,
Peter Hansen <peter at engcorp.com> wrote:
>Rafael Mentz Aquino wrote:
>> How many lines can I have in a simple ascii file before
>> I start to loose performance when I open this file and
>> put all the lines in a dict, turning the first column into the key
>> and the rest of the line into a list associated to that key.
>> (my file has just 100 lines, but that question starts to bother me ;-)
>You will start to lose performance, relatively to no lines,
>with even one line. If you start with ten lines, you will
>begin losing performance after you switch to a file of 11
>Why focus on performance when you have no idea how fast
>you want your program to be? Recent advances in software
>development are teaching programmers *never* to optimize
>until the program works, and only when there is a clear
>requirement for more performance.
>If you are really concerned about performance, you will
>always be able to get more than a 10 times speedup by
>switching to a compiled language over Python. Use of
>Python should imply you are more interested in readability
>and ease of use, and thus speed of *development*, over
>speed of execution.
>(Shorter answer: don't grow any grey hairs over it.)
Peter and I agree on so many large conclusions,
and differ on so many details.
My own suspicion about the situation the original
poster describes is that I/O time dominates, and
that neither "switching to a compiled language" nor
a fancier persistence method will help by even a
factor of two (maybe not 10%) in comparison with
judiciously straightforward Python coding. I know
that's true for many things I do: it just takes a
long time to read in data. Once it's in memory,
Python manages it plenty swiftly.
Peter's entirely correct in his general advice.
Cameron Laird <Cameron at Lairds.com>
More information about the Python-list