pickle: huge memory consumption *during* pickling
Hans Georg Krauthaeuser
hgk at et.uni-magdeburg.de
Fri Nov 12 08:45:23 CET 2004
Yes, I'm sure that I'm using cPickle.
But, I don't use protocol 2. I will try that and post the difference.
Thanks for the hint.
Jean Brouwers wrote:
> Good point. Double check first that you use
> - import cPickle
> - cPickle.dump(<obj>, <file>, 2) # note, protocol 2
> /Jean Brouwers
> ProphICy Semiconductor, Inc.
> In article <mailman.6272.1100201176.5135.python-list at python.org>,
> Gerrit <gerrit at nl.linux.org> wrote:
>>Hans Georg Krauthaeuser wrote:
>>>Today morning we reached the situation that it took 6 hours to pickle
>>>the class instance. The pickle file was than approx. 92 MB (this is ok).
>>>During pickling the memory consuption of the python proccess was up to
>>>450 MB (512 MB RAM -> machine was swapping all the time).
>>>My class use data types taken from a c++ class via swig. Don't know if
>>>that is important...
>>>My feeling is that I'm doing something wrong. But my python knowlegde is
>>>not so deep to see what that is.
>>>Is there an other way to perform an autosave of an class instance? Shelve?
>>Are you sure you are using cPickle as opposed to pickle?
More information about the Python-list