pickle.load() extremely slow performance
jhg at acm.org
Fri Mar 20 23:25:37 CET 2009
I'm converting a Perl system to Python, and have run into a severe
performance problem with pickle.
One facet of the system involves scanning and loading into memory a
couple of parallel directory trees containing OTO 10^4 files. The
trees don't change during development/testing and the scan takes 30-40
seconds, so to save time I cache the loaded tree structure to disk, in
Perl with module Storable, and in Python with pickle.
In Perl, the save operation produces a file of about 3MB, and both
save and restore take a second or two. In Python, pickle.dump()
produces a similar-size file but takes 20 seconds, and pickle.load()
takes 45 seconds, which is actually LONGER than the time required to
scan the directory trees.
Is there anything I can do to speed up pickle.load() to get
performance comparable to Perl's Storable?
More information about the Python-list