pickle: huge memory consumption *during* pickling

[Tony Clarke]

Hans Georg Krauthaeuser <hgk at et.uni-magdeburg.de> wrote in message news:<cn00o5$r0j$1 at fuerst.cs.uni-magdeburg.de>...
> Dear all,
> 
> I have a long running application (electromagnetic compatibility 
> measurements in mode-stirred chambers over GPIB) that use pickle 
> (cPickle) to autosave a class instance with all the measured data from 
> time to time.
> 
> At the beginning, pickling is quite fast but when the data becomes more 
> and more pickling slows down rapidly.
>(Snip)
> My feeling is that I'm doing something wrong. But my python knowlegde is 
> not so deep to see what that is.
> 
> Is there an other way to perform an autosave of an class instance? Shelve?
> 
> System: python 2.3.4, Win XP, 1.X GHz class PC, 512 MB ram
>
> Best redards
> Hans Georg

The tutorial books I read (including the Python Bible, I think) said
that pickle shouldn't be used for large objects, so I try to limit it
to smaller objects in small applications. I always wondered what they
meant by large objects, maybe this is an illustration of that? :)

Would it not be possible to save your data as a file, (or use a class 
method to download the stored data to a file) on your disc?  You could
always reload it from there for further use. Or split the class into
several smaller ones, each of which might be more efficient at using
pickle?

Regards 

Tony Clarke