memory leak with large list??

[Terry Reedy]

> Then I generate a large list (12 million floats), either using map,
a
> list comprehension, or preallocating a list, then filling it using a
> for loop.

12 million different floats use 12 million * sizeof float object
bytes. Float object is at least 8 bytes for float + object overhead
(about 4 bytes?)  list has array of 12 millions pointers (4 (or
possibly 8) bytes).  Thus at least  be 16 * 12 million, roughly 200
million bytes, regardless of method.

...

> When I check the memory footprint again, it is larger than I would
> expect, but I'm not sure what the per element overhead of a list is.
>
> %MEM  RSS  RSS    SZ   VSZ
> 46.2 237648 237648 59690 238760

If 237648 is # of Kbytes, it is roughly what should expect.

> So then delete it.
>
> >>> del(big_list)
>
> But the memory footprint of the process remains very large:
>
> %MEM  RSS  RSS    SZ   VSZ
> 37.1 190772 190772 47971 191884

Deleting frees mem for reuse by Python but does not necessarily return
memory to system for reuse by other processes.  (Exact behavior is
*very* system specific, according to Tim Peters' war stories.)

> Is python leaking memory, or is there something else going on here.

Pretty sure no, yes.

> I should note, if I just preallocate the list with 0.0:
>
> >>> big_list = [0.0] * 12000000

(I presume you mean, by preallocate, that you subsequently overwrote
with floats as before.)  This is 12 million duplicate pointers to
*one* object.  48 megabyte memory for array is allocated just once.
Others methods start with small list (perhaps with room for 8
pointers) and reallocate and copy as list grows.  This can chop up
heap.  Preallocation is much better for large arrays.

> and then delete it, I don't see the leak, but then memory footprint
is
> smaller than you would expect
> from 12 million floats:
>
> %MEM  RSS  RSS    SZ   VSZ
>  9.4 48700 48700 12453 49812
> So I think python is using a shortcut in this case.
> I am using Python 2.2.1 on Debian 3.0, kernel 2.4.17-686.

Maybe someone else is familiar with this particular system.

Terry J. Reedy