pickle: huge memory consumption *during* pickling

Gerrit gerrit at nl.linux.org
Thu Nov 11 20:25:36 CET 2004

Hans Georg Krauthaeuser wrote:
> Today morning we reached the situation that it took 6 hours to pickle 
> the class instance. The pickle file was than approx. 92 MB (this is ok). 
> During pickling the memory consuption of the python proccess was up to 
> 450 MB (512 MB RAM -> machine was swapping all the time).
> My class use data types taken from a c++ class via swig. Don't know if 
> that is important...
> My feeling is that I'm doing something wrong. But my python knowlegde is 
> not so deep to see what that is.
> Is there an other way to perform an autosave of an class instance? Shelve?

Are you sure you are using cPickle as opposed to pickle?

Gerrit Holl.

Weather in Twenthe, Netherlands 11/11 19:25:
	-1.0°C   wind 0.4 m/s None (57 m above NAP)
In the councils of government, we must guard against the acquisition of
unwarranted influence, whether sought or unsought, by the
military-industrial complex. The potential for the disastrous rise of
misplaced power exists and will persist.
    -Dwight David Eisenhower, January 17, 1961

More information about the Python-list mailing list