pickle: huge memory consumption *during* pickling

[Hans Georg Krauthaeuser]

Yes, I'm sure that I'm using cPickle.

But, I don't use protocol 2. I will try that and post the difference.

Thanks for the hint.

Hans Georg

Jean Brouwers wrote:
> Good point.  Double check first that you use
> 
> - import cPickle
> 
> - cPickle.dump(<obj>, <file>, 2)  # note, protocol 2
> 
> /Jean Brouwers
>   ProphICy Semiconductor, Inc.
> 
> 
> 
> In article <mailman.6272.1100201176.5135.python-list at python.org>,
> Gerrit <gerrit at nl.linux.org> wrote:
> 
> 
>>Hans Georg Krauthaeuser wrote:
>>
>>>Today morning we reached the situation that it took 6 hours to pickle 
>>>the class instance. The pickle file was than approx. 92 MB (this is ok). 
>>>During pickling the memory consuption of the python proccess was up to 
>>>450 MB (512 MB RAM -> machine was swapping all the time).
>>>
>>>My class use data types taken from a c++ class via swig. Don't know if 
>>>that is important...
>>>
>>>My feeling is that I'm doing something wrong. But my python knowlegde is 
>>>not so deep to see what that is.
>>>
>>>Is there an other way to perform an autosave of an class instance? Shelve?
>>
>>Are you sure you are using cPickle as opposed to pickle?
>>
>>regards,
>>Gerrit Holl.