handeling very large dictionaries

Joseph Turian turian at gmail.com
Mon Jun 29 01:54:57 EDT 2009


You could also try using a key-value store.
I am using pytc, a Python API for Tokyo Cabinet. It seems to figure
out quite nicely when to go to disk, and when to use memory. But I
have not done extensive tests.

Here is some example code for using pytc:
  http://github.com/turian/pytc-example/tree/master

 Joseph

On Jun 28, 7:13 pm, mclovin <hanoo... at gmail.com> wrote:
> Hello all,
>
> I need to have a dictionary of about 8 gigs (well the data it is
> processing is around 4gb). so naturally i am running into memory
> errors.
>
> So i looked around and found bsddb which acts like a dictionary object
> only offloads the data from the RAM to the HDD, however that only
> supports strings.
>
> my dictionaries hold my own class objects
>
> Is there something like it that is more flexible?




More information about the Python-list mailing list