Dictionary or Database—Please advise

[Paul Rubin]

Jeremy <jlconlin at gmail.com> writes:
> I have lots of data that I currently store in dictionaries.  However,
> the memory requirements are becoming a problem.  I am considering
> using a database of some sorts instead, but I have never used them
> before.  Would a database be more memory efficient than a dictionary?

What are you trying to do?  Yes, a database would let you use disk
instead of ram, but the slowdown might be intolerable depending on the
access pattern.  You really have to consider what the whole application
is up to, and what the database is doing for you.  It might be imposing
a high overhead to create benefits (e.g. transaction isolation) that
you're not actually using.

Somehow I've managed to do a lot of programming on large datasets
without ever using databases very much.  I'm not sure that's a good
thing; but anyway, a lot of times you can do better with externally
sorted flat files, near-line search engines, etc. than you can with
databases.

If the size of your data is fixed or is not growing too fast, and it's
larger than your computer's memory by just a moderate amount (e.g. you
have a 2GB computer and 3GB of data), the simplest approach may be to
just buy a few more ram modules and put them in your computer.