key/value store optimized for disk storage

[Terry Reedy]

On 5/2/2012 10:14 PM, Steve Howell wrote:
> This is slightly off topic, but I'm hoping folks can point me in the
> right direction.
>
> I'm looking for a fairly lightweight key/value store that works for
> this type of problem:
>
>    ideally plays nice with the Python ecosystem
>    the data set is static, and written infrequently enough that I
> definitely want *read* performance to trump all
>    there is too much data to keep it all in memory (so no memcache)
>    users will access keys with fairly uniform, random probability
>    the key/value pairs are fairly homogenous in nature:
>      keys are<= 16 chars
>      values are between 1k and 4k bytes generally
>    approx 3 million key/value pairs
>    total amount of data == 6Gb
>    needs to work on relatively recent versions of FreeBSD and Linux

On my 64bit machine with 64 bit Python, I would consider putting all the 
values in one data file and creating a key:file-offset dict. Each value 
would start with length(value) so that is not needed in memory. The 
dict, once created, could be pickled and unpickled for each run.

-- 
Terry Jan Reedy