[Tutor] numerical simulation + SQLite

[Alan Gauld]

"Faisal Moledina" <faisal.moledina at gmail.com> wrote 
.
> At first, I used a list to store a bunch of class instances, each of
> which contained a bunch of data calculated at each time step. This
> resulted in whopping memory usage (2.75 GB RAM, 3.75 GB VM).
> 
> So then I decided instead to use SQLite to store that information at
> each timestep. This seems to work well, but it gets slow over time as
> well. 

You may need to be realistic in your expectations.
A database is writing to disk which will be slower than working in 
memory. And a 3GB file takes a while to read/traverse, even with 
indexes. It depends a lot on exactly what you are doing. If its mainly 
writing it should not be much slower than writing to a flat file. If you 
are doing a lot of reading - and you have used indexes - then it 
should be a lot faster than a file.

But RAM - if you have enough - will always be fastest, by about 100 times.
The problem is when you run out, you revert to using files and that's 
usually slower than a database...

But without details of your usage pattern and database schema 
and SQL code etc it is, as you say, impossible to be specific.

-- 
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/