Re: Dictionary or Database—Please advise

[Steven D'Aprano]

On Fri, 26 Feb 2010 21:56:47 +0100, Patrick Sabin wrote:

>> Shelve looks like an interesting option, but what might pose an issue
>> is that I'm reading the data from a disk instead of memory.  I didn't
>> mention this in my original post, but I was hoping that by using a
>> database it would be more memory efficient in storing data in RAM so I
>> wouldn't have to read from (or swap to/from) disk.
> 
> A database usually stores data on disk and not in RAM. However you could
> use sqlite with :memory:, so that it runs in RAM.

The OP started this thread with:

"I have lots of data that I currently store in dictionaries.  However,
the memory requirements are becoming a problem."

So I'm amused that he thinks the solution to running out of memory is to 
run the heavy overhead of a database entirely in memory, instead of 
lightweight dicts :)

My advice is, first try to optimize your use of dicts. Are you holding 
onto large numbers of big dicts that you don't need? Are you making 
unnecessary copies? If so, fix your algorithms.

If you can't optimize the dicts anymore, then move to a proper database. 
Don't worry about whether it is on-disk or not, databases tend to be 
pretty fast regardless, and it is better to run your application a little 
bit more slowly than to not run it at all because it ran out of memory 
halfway through processing the data.


-- 
Steven