writing large dictionaries to file using cPickle

Aaron Brady castironpi at gmail.com
Wed Jan 28 22:45:19 CET 2009

On Jan 28, 10:13 am, perfr... at gmail.com wrote:
> hello all,
> i have a large dictionary which contains about 10 keys, each key has a
> value which is a list containing about 1 to 5 million (small)
> dictionaries. for example,
> but this takes just as long... any ideas ? is there a different module
> i could use that's more suitable for large dictionaries ?
> thank you very much.

There is the 'shelve' module.  You could create a shelf that tells you
the filename of the 5 other ones.  A million keys should be no
problem, I guess.  (It's standard library.)  All your keys have to be
strings, though, and all your values have to be pickleable.  If that's
a problem, yes you will need ZODB or Django (I understand), or another
relational DB.

More information about the Python-list mailing list