Memory sizes of python objects?

[Martin v. Loewis]

"Clark C . Evans" <cce at clarkevans.com> writes:

> Hello.  I'm trying to figure out how much memory overhead
> is used with using strings, tuples, lists, and maps.
> In particular, say I have a nested structure like...

To find out such things, I recommend to study the source code of
Python, in particular the header files. To see such a computation,
please read

http://groups.google.de/groups?hl=de&selm=j43d3m93iu.fsf%40informatik.hu-berlin.de

> etc.  Is there some rule of thumb that I can use to estimate,
> for example, take the character data you have and multiply by
> a factor of 4 to find the in-memory footprint of tuples, and
> each map is 1K plus 64 bytes per entry..

For a rule-of-thumb, you should be aware of the per-object overhead,
and the pointer size on your system. The per-object overhead consists
of three pieces (assuming a 32-bit system):

- the Python per-object overhead: 8 bytes for fixed size objects
  (e.g. integers), and 12 bytes for variable-sized objects (strings,
  tuples)

- the garbage collector overhead: 8 bytes for containers
  (e.g. tuples), nothing for non-containers (strings)

- the malloc overhead: varies widely by platform, but it is 8 bytes in
  most cases. In addition, malloc will usually round up the object
  size to a multiple of 8.

In addition, you have the per-content overhead, which varies with the
type of object:

- 4 bytes for an integer
- 1 byte per character in a string, plus one for the terminating 0
- 4 bytes per element in a list or tuple (notice that lists allocate
  space for more elements in advance)

Dictionaries are more difficult to count; their size also varies with
the Python version. The dictionary object itself is one memory block,
including gc and object overhead (not counting the malloc overhead),
the dictionary takes 144 bytes (in Python 2.2). The array of entries
is another memory block, which takes 12 bytes per entry.

Notice that dictionaries overallocate entries, so you'll always need
space for more entries than you have in the dictionary. Also, "small
dictionaries" (less than 9 entries) don't need any extra space, since
they store their entries in the 144 bytes.

Regards,
Martin