memory efficient set/dictionary

[Josiah Carlson]

Rafael Darder Calvo wrote:
>> > > Please recommend a module that allows persistent set/dict storage +
>> > > fast query that best fits my problem,
>> >
>> > What is the problem you are trying to solve? How many keys do you have?
>>
>> Corpus processing. There are in the order of billions to tens of
>> billions keys (64bit integers).
>>
> I would recommend you to use a database since it meets your
> requirements (off-memory, fast, persistent). The bsdddb module
> (berkeley db) even gives you a dictionary like interface.
> http://www.python.org/doc/lib/module-bsddb.html

Standard SQL databases can work for this, but generally your 
recommendation of using bsddb works very well for int -> int mappings. 
In particular, I would suggest using a btree, if only because I have had 
troubles in the past with colliding keys in the bsddb.hash (and recno is 
just a flat file, and will attempt to create a file i*(record size) to 
write to record number i .

As an alternative, there are many search-engine known methods for 
mapping int -> [int, int, ...], which can be implemented as int -> int, 
where the second int is a pointer to an address on disk.  Looking into a 
few of the open source search implementations may be worthwhile.

  - Josiah