performance of pickling & large lists :-(
Andreas.Leitgeb at siemens.at
Tue Aug 6 20:23:09 CEST 2002
Being new in Python, I try lots of things, whether actually
useful or not :-)
I had a really large list (of about 4 million entries) of lists of
each 10 int's. The process had a footprint of about 300MB then.
Then I tried to pickle it to a file (not to string :-). I'd have
expected, that while the file would become quite large, hardly any
additional memory would be needed, as the pickler would iterate through
the elements, create a reversible string-representation for each
and write it out, then throw away these temporary strings.
But during pickling the memory-consumption was increasing and
increasing. I finally stopped it, when the memory-footprint had
grown large enough to get the machine into constant swapping.
(the machine has 320MB ram in it, Python grew to about 344MB before
I killed it. At that time not even a tenth of the whole list had been
Is pickling more complicated internally, than I expected it to be ?
when I create a few objects , (just enough to see memory consumption go
up for python in "top"), and then release them ("del"), then I see memory
consumption go down (almost) to the old size.
But, when I create a really large list, and del it, then the memory-
footprint remains large.
Is this a bug or a feature ?
(seen with 2.2.0, not yet tried with others)
Newsflash: Sproingy made it to the ground !
read more ... <http://avl.enemy.org/sproingy/>
More information about the Python-list