is cPickle really this slow?
dag at animagicnet.no
Mon Sep 30 05:53:21 EDT 2002
I have a script which starts by parsing a large file and turning into
a data stucture (a dictonary of lists) which I then use in the script.
This operation however takes about 12-15 seconds. Since the file hardly
ever changes , and if it changes it's because I changed it, thought I'd
save some time by pickeling the datastructure and and simply getting that
at the start of each script. So I pickle my dictionary, which ends up
being a 3.2MB file and try to run my script again, this time getting
the pickled dictonary instead of building it at runtime. This time
however the script takes over 35 seconds to run, close to three times
as long. Am I doing something wrong is loading and parsing larger
pickled files really very slow.
More information about the Python-list