key/value store optimized for disk storage

[Tim Chase]

On 05/02/12 21:14, Steve Howell wrote:
> I'm looking for a fairly lightweight key/value store that works for
> this type of problem:
> 
>   ideally plays nice with the Python ecosystem
>   the data set is static, and written infrequently enough that I
> definitely want *read* performance to trump all
>   there is too much data to keep it all in memory (so no memcache)
>   users will access keys with fairly uniform, random probability
>   the key/value pairs are fairly homogenous in nature:
>     keys are <= 16 chars
>     values are between 1k and 4k bytes generally
>   approx 3 million key/value pairs
>   total amount of data == 6Gb
>   needs to work on relatively recent versions of FreeBSD and Linux
[snip]
> I'm looking for something a little lighter than a full-on database
> (either SQL or no-SQL), although I'm not completely ruling out any
> alternatives yet.

This sounds suspiciously like the standard library's anydbm
module[1] which should handle everything you list (though I can't be
positive about the 6Gb bit, though I expect it should be fine; I
can't find any documentation on the limits).

-tkc

[1]
http://docs.python.org/library/anydbm.html