optimizing memory utilization
ialbert at mailblocks.com
Wed Sep 15 17:39:22 CEST 2004
> can't believe that python requires memory 200% "overhead" to store my
> data as lists of lists.
And it almost certainly doesn't.
One one hand there could be a sizebale overhead when running
your program even when loading a single data line.
So don't extrapolate from a small datasets.
But what I believe happens to you is that you keep references
alive and you end up with data duplication.
For example if you do:
lines = file('whatever').readlines()
then you say for example:
stripped_lines = [ line.strip() for line in lines ]
you've just made your program use twice the memory
at that point of the code.
You may also inadvertently keep your 'lines' variable
in the scope of the full program then
you'll effectively end up with a lot more memory consumption
Check and plug all these holes.
More information about the Python-list