Why custom objects take so much memory?

[Chris Mellon]

On Dec 18, 2007 1:26 PM, jsanshef <jsanpedro at gmail.com> wrote:
> Hi,
>
> after a couple of days of script debugging, I kind of found that some
> assumptions I was doing about the memory complexity of my classes are
> not true. I decided to do a simple script to isolate the problem:
>
> class MyClass:
>         def __init__(self,s):
>                 self.mystring  = s
>
> mylist = []
> for i in range(1024*1024):
>         mylist.append(MyClass(str(i))) #allocation
> #stage 1
> mylist = None
> gc.collect()
> #stage 2
>
> I take measures of the memory consumption of the script at #stage1 and
> #stage 2 and I obtain:
> #stage1 -> 238MB
> #stage2 -> 15MB
>
> That means every object is around 223 bytes in size!!!! That's too
> much considering it only contains a string with a maximum size of 7
> chars.
>

Classes are fairly heavyweight - in your case you've got the size of
the PyObject struct, a dictionary for class attributes (which itself
is another pyobject), and the string object (yet another pyobject),
and the actual string data.

> If you change the allocation line for this other:
> >>mylist.append(str(i)) #we don't create the custom class, but append the string directly into the list
>
> the numbers decrease substantially to:
> #stage1 -> 47.6MB
> #stage2 -> 15MB
> (so this time we can say string vars occupy around 32 bytes....still a
> lot, isn't it?)
>

string objects don't have dictionaries and are smaller than "regular"
python objects.

> So, what's exactly going on behind the scenes? Why is using custom
> objects SO expensive? What other ways of creating structures can be
> used (cheaper in memory usage)?
>

If you're worried about per-instance memory costs Python is probably
not the language for your purposes. On the other hand, odds are that
you actually don't need to worry so much.

You can reduce the size of new-style classes (inherit from object) by
quite a bit if you use __slots__ to eliminate the class dictionary.