writing large dictionaries to file using cPickle

repi repi8 at hotmail.com
Sun Apr 26 23:12:32 CEST 2009


I'm having the same problem. I'm creating two dictionaries with about
700,000 to 1,600,000 keys each, each key pointing to a smaller structure of
embedded dictionaries. On a 8 x 3.0GHz Xeon Mac Pro with 17GB of RAM,
cPickle is excruciatingly slow. Moreover, it crashes with memory error (12).
I've been looking for alternatives, even thinking of writing up my own file
format and read/write functions. As a last resort I tried dumping the two
lists with the marshal module and it worked great. A 200MB and a 300MB file
are created in a couple of seconds without a crash. 

You should be warned that marshal is not meant as a general data-persistence
module. But as long as you have a rather simple data structure and you are
using it with a specific python version, it seems to run circles around

Hope it helps.

View this message in context: http://www.nabble.com/writing-large-dictionaries-to-file-using-cPickle-tp21708938p23246693.html
Sent from the Python - python-list mailing list archive at Nabble.com.

More information about the Python-list mailing list