Memory issues when storing as List of Strings vs List of List

[Antoine Pitrou]

On Tue, 30 Nov 2010 18:29:35 +0800
OW Ghim Siong <owgs at bii.a-star.edu.sg> wrote:
> 
> Does anyone know why is there such a big difference memory usage when 
> storing the matrix as a list of list, and when storing it as a list of 
> string?

That's because any object has a fixed overhead (related to metadata and
allocation), so storing a matrix line as a sequence of several objects
rather than a single string makes the total overhead larger,
especially when the payload of each object is small.

If you want to mitigate the issue, you could store your lines as tuples
rather than lists, since tuples have a smaller memory footprint:

    matrix.append(tuple(data))

> According to __sizeof__ though, the values are the same whether 
> storing it as a list of list, or storing it as a list of string.

As mentioned by others, __sizeof__ only gives you the size of the
container, not the size of the contained values (which is where the
difference is here).

Regards

Antoine.