Re: Dictionary or Database—Please advise
steve at REMOVE-THIS-cybersource.com.au
Sat Feb 27 03:21:04 CET 2010
On Fri, 26 Feb 2010 21:56:47 +0100, Patrick Sabin wrote:
>> Shelve looks like an interesting option, but what might pose an issue
>> is that I'm reading the data from a disk instead of memory. I didn't
>> mention this in my original post, but I was hoping that by using a
>> database it would be more memory efficient in storing data in RAM so I
>> wouldn't have to read from (or swap to/from) disk.
> A database usually stores data on disk and not in RAM. However you could
> use sqlite with :memory:, so that it runs in RAM.
The OP started this thread with:
"I have lots of data that I currently store in dictionaries. However,
the memory requirements are becoming a problem."
So I'm amused that he thinks the solution to running out of memory is to
run the heavy overhead of a database entirely in memory, instead of
lightweight dicts :)
My advice is, first try to optimize your use of dicts. Are you holding
onto large numbers of big dicts that you don't need? Are you making
unnecessary copies? If so, fix your algorithms.
If you can't optimize the dicts anymore, then move to a proper database.
Don't worry about whether it is on-disk or not, databases tend to be
pretty fast regardless, and it is better to run your application a little
bit more slowly than to not run it at all because it ran out of memory
halfway through processing the data.
More information about the Python-list