Memory efficient tuple storage

psaffrey at googlemail.com psaffrey at googlemail.com
Fri Mar 13 16:59:17 CET 2009


I'm reading in some rather large files (28 files each of 130MB). Each
file is a genome coordinate (chromosome (string) and position (int))
and a data point (float). I want to read these into a list of
coordinates (each a tuple of (chromosome, position)) and a list of
data points.

This has taught me that Python lists are not memory efficient, because
if I use lists it gets through 100MB a second until it hits the swap
space and I have 8GB physical memory in this machine. I can use Python
or numpy arrays for the data points, which is much more manageable.
However, I still need the coordinates. If I don't keep them in a list,
where can I keep them?

Peter



More information about the Python-list mailing list