pickle: huge memory consumption *during* pickling

[Nick Craig-Wood]

Hans Georg Krauthaeuser <hgk at et.uni-magdeburg.de> wrote:
>  I have a long running application (electromagnetic compatibility 
>  measurements in mode-stirred chambers over GPIB) that use pickle 
>  (cPickle) to autosave a class instance with all the measured data from 
>  time to time.
> 
>  At the beginning, pickling is quite fast but when the data becomes more 
>  and more pickling slows down rapidly.
> 
>  Today morning we reached the situation that it took 6 hours to pickle 
>  the class instance. The pickle file was than approx. 92 MB (this is ok). 
>  During pickling the memory consuption of the python proccess was up to 
>  450 MB (512 MB RAM -> machine was swapping all the time).

You've probably got lots of instances of a single class...  We managed
to 1/3 the memory requirments in a similar situation by using new
style classes (inherit from object) and defining __slots__ for just a
single class!

>  My class use data types taken from a c++ class via swig. Don't know if 
>  that is important...

This may be important I don't know!

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick