key/value store optimized for disk storage

[William R. Wing (Bill Wing)]

On May 2, 2012, at 10:14 PM, Steve Howell wrote:

> This is slightly off topic, but I'm hoping folks can point me in the
> right direction.
> 
> I'm looking for a fairly lightweight key/value store that works for
> this type of problem:
> 
>  ideally plays nice with the Python ecosystem
>  the data set is static, and written infrequently enough that I
> definitely want *read* performance to trump all
>  there is too much data to keep it all in memory (so no memcache)
>  users will access keys with fairly uniform, random probability
>  the key/value pairs are fairly homogenous in nature:
>    keys are <= 16 chars
>    values are between 1k and 4k bytes generally
>  approx 3 million key/value pairs
>  total amount of data == 6Gb
>  needs to work on relatively recent versions of FreeBSD and Linux
> 


I don't understand that statement, I don't think I agree with it.  I'm writing this e-mail on a system with 16 GB of memory.  It is just under 3 years old, and if it were purchased today, it would have 32 GB of memory at essentially the same price.

I think you might want to re-examine your thinking about the limits of a memory-based system.  It could probably be completely written in python and still have acceptable performance.

-Bill