writing large dictionaries to file using cPickle

[repi]

Hello,

I'm having the same problem. I'm creating two dictionaries with about
700,000 to 1,600,000 keys each, each key pointing to a smaller structure of
embedded dictionaries. On a 8 x 3.0GHz Xeon Mac Pro with 17GB of RAM,
cPickle is excruciatingly slow. Moreover, it crashes with memory error (12).
I've been looking for alternatives, even thinking of writing up my own file
format and read/write functions. As a last resort I tried dumping the two
lists with the marshal module and it worked great. A 200MB and a 300MB file
are created in a couple of seconds without a crash. 

You should be warned that marshal is not meant as a general data-persistence
module. But as long as you have a rather simple data structure and you are
using it with a specific python version, it seems to run circles around
cPickle.

Hope it helps.

-- 
View this message in context: http://www.nabble.com/writing-large-dictionaries-to-file-using-cPickle-tp21708938p23246693.html
Sent from the Python - python-list mailing list archive at Nabble.com.