[Tutor] numerical simulation + SQLite
alan.gauld at btinternet.com
Mon Nov 30 23:05:50 CET 2009
"Faisal Moledina" <faisal.moledina at gmail.com> wrote
> At first, I used a list to store a bunch of class instances, each of
> which contained a bunch of data calculated at each time step. This
> resulted in whopping memory usage (2.75 GB RAM, 3.75 GB VM).
> So then I decided instead to use SQLite to store that information at
> each timestep. This seems to work well, but it gets slow over time as
You may need to be realistic in your expectations.
A database is writing to disk which will be slower than working in
memory. And a 3GB file takes a while to read/traverse, even with
indexes. It depends a lot on exactly what you are doing. If its mainly
writing it should not be much slower than writing to a flat file. If you
are doing a lot of reading - and you have used indexes - then it
should be a lot faster than a file.
But RAM - if you have enough - will always be fastest, by about 100 times.
The problem is when you run out, you revert to using files and that's
usually slower than a database...
But without details of your usage pattern and database schema
and SQL code etc it is, as you say, impossible to be specific.
Author of the Learn to Program web site
More information about the Tutor