Pickle MemoryError - any ideas?

John Nagle nagle at animats.com
Tue Jul 20 20:58:03 EDT 2010


On 7/20/2010 3:01 PM, Peter wrote:
> I have created a class that contains a list of files (contents,
> binary) - so it uses a LOT of memory.
>
> When I first pickle.dump the list it creates a 1.9GByte file on the
> disk. I can load the contents back again, but when I attempt to dump
> it again (with or without additions), I get the following:

    Be sure to destroy the pickle object when you're done with it.
Don't reuse it. Pickle has a cache - it saves every object pickled, and
if the same object shows up more than once, the later instances
are represented as a cache ID.  This can fill memory unnecessarily.

    See 
"http://groups.google.com/group/comp.lang.python/browse_thread/thread/3f8b999c25af263a"

				John Nagle



More information about the Python-list mailing list