cPickle asymptotic performance?
jonas at MIT.EDU
Thu Jun 12 23:21:31 CEST 2008
On Thu, 2008-06-12 at 20:57 +0200, Hrvoje Niksic wrote:
> Eric Jonas <jonas at MIT.EDU> writes:
> > I've done some benchmarking while attempting to serialize my (large)
> > graph data structure with cPickle; I'm seeing superlinear performance
> > (plotting it seems to suggest n^2 where n is the number of nodes of my
> > graph), in the duration of the pickle.dump calls and I can't quite
> > figure out why.
> Try gc.disable() before loading the pickle, and gc.enable() after.
> > Is cPickle's behavior known to be O(n^2)?
> No, but the garbage collector's sometimes is.
Wow, that totally fixed it -- we went from 1200 seconds to 60 seconds.
I'm somewhat shocked -- is the occasionally-abysmal behavior of the GC
More information about the Python-list