is cPickle really this slow?
daniel.dittmar at sap.com
Mon Sep 30 07:29:11 EDT 2002
> I have a script which starts by parsing a large file and turning into
> a data stucture (a dictonary of lists) which I then use in the script.
> This operation however takes about 12-15 seconds. Since the file
> hardly ever changes , and if it changes it's because I changed it,
> thought I'd save some time by pickeling the datastructure and and
> simply getting that at the start of each script. So I pickle my
> dictionary, which ends up being a 3.2MB file and try to run my script
> again, this time getting the pickled dictonary instead of building it
> at runtime. This time however the script takes over 35 seconds to
> run, close to three times as long. Am I doing something wrong is
> loading and parsing larger pickled files really very slow.
Have you used the binary option to 'dump'?
dump(object, file[, bin]) : If the optional bin argument is true, the binary
pickle format is used; otherwise the (less efficient) text pickle format is
used (for backwards compatibility, this is the default).
More information about the Python-list