is cPickle really this slow?

[Daniel Dittmar]

Dag wrote:
> I have a script which starts by parsing a large file and turning into
> a data stucture (a dictonary of lists) which I then use in the script.
> This operation however takes about 12-15 seconds.  Since the file
> hardly ever changes , and if it changes it's because I changed it,
> thought I'd save some time by pickeling the datastructure and and
> simply getting that at the start of each script.  So I pickle my
> dictionary, which ends up being a 3.2MB file and try to run my script
> again, this time getting the pickled dictonary instead of building it
> at runtime.  This time however the script takes over 35 seconds to
> run, close to three times as long.  Am I doing something wrong is
> loading and parsing larger pickled files really very slow.

Have you used the binary option to 'dump'?
dump(object, file[, bin]) : If the optional bin argument is true, the binary
pickle format is used; otherwise the (less efficient) text pickle format is
used (for backwards compatibility, this is the default).

Daniel