[Tutor] numerical simulation + SQLite

Alan Gauld alan.gauld at btinternet.com
Mon Nov 30 23:05:50 CET 2009

"Faisal Moledina" <faisal.moledina at gmail.com> wrote 
> At first, I used a list to store a bunch of class instances, each of
> which contained a bunch of data calculated at each time step. This
> resulted in whopping memory usage (2.75 GB RAM, 3.75 GB VM).
> So then I decided instead to use SQLite to store that information at
> each timestep. This seems to work well, but it gets slow over time as
> well. 

You may need to be realistic in your expectations.
A database is writing to disk which will be slower than working in 
memory. And a 3GB file takes a while to read/traverse, even with 
indexes. It depends a lot on exactly what you are doing. If its mainly 
writing it should not be much slower than writing to a flat file. If you 
are doing a lot of reading - and you have used indexes - then it 
should be a lot faster than a file.

But RAM - if you have enough - will always be fastest, by about 100 times.
The problem is when you run out, you revert to using files and that's 
usually slower than a database...

But without details of your usage pattern and database schema 
and SQL code etc it is, as you say, impossible to be specific.

Alan Gauld
Author of the Learn to Program web site

More information about the Tutor mailing list