pickle: huge memory consumption *during* pickling
Robert Brewer
fumanchu at amor.org
Thu Nov 11 12:54:15 EST 2004
Hans Georg Krauthaeuser wrote:
> I have a long running application (electromagnetic compatibility
> measurements in mode-stirred chambers over GPIB) that use pickle
> (cPickle) to autosave a class instance with all the measured
> data from
> time to time.
>
> At the beginning, pickling is quite fast but when the data
> becomes more
> and more pickling slows down rapidly.
>
> Today morning we reached the situation that it took 6 hours to pickle
> the class instance. The pickle file was than approx. 92 MB
> (this is ok).
> During pickling the memory consuption of the python proccess
> was up to
> 450 MB (512 MB RAM -> machine was swapping all the time).
>
> My class use data types taken from a c++ class via swig.
> Don't know if
> that is important...
>
> My feeling is that I'm doing something wrong. But my python
> knowlegde is
> not so deep to see what that is.
>
> Is there an other way to perform an autosave of an class
> instance? Shelve?
Shelve just uses pickle:
try:
from cPickle import Pickler, Unpickler
except ImportError:
from pickle import Pickler, Unpickler
...but it might help if the bottleneck is in your file-handling.
Robert Brewer
MIS
Amor Ministries
fumanchu at amor.org
More information about the Python-list
mailing list