memory freeing

[Jonathan Hogg]

On 21/7/2002 17:08, in article
ced73313.0207210808.23b585b9 at posting.google.com, "Peter Saffrey"
<theoryboy at my-deja.com> wrote:

> I set up a loop to assign "None" values into all the reachable_from
> hashes, but the memory displayed by my system monitor (gkrellm) does
> not seem to indicate that I've gained any memory. Do I need to do
> anything more specific to order Python to return this memory to me?
> Obviously many of the routes have shared parts, but I use the
> concatenate (+) operator when building the lists, and I thought this
> generated new list copies. Apart from anything else, each hash and the
> pointers to the lists of lists should be separate objects, and just
> removing these ought to give me a few megabytes, but I don't seem to
> be getting any at all.

Assuming the objects really are being deallocated (a good way to test this
with a recent Python with garbage collection is:

>>> import gc
>>> len( gc.get_objects() )
1223
>>> foo = [(x,y) for x in range(100) for y in range(100)]
>>> len( gc.get_objects() )
11224
>>> del foo
>>> len( gc.get_objects() )
1223
>>> 

which shows the number of allocated container objects) then the answer is
probably that your particular combination of pymalloc/malloc/sbrk/whatever
is not necessarily returning the memory to the OS.

This doesn't mean that it's not been deallocated. There's a whole
complicated hierarchy of memory allocation mechanisms in play (object
freelists, pymalloc, malloc, etc.) and the memory might not have bubbled all
the way back. You'll likely find that it has been returned to your Python
program so you can allocate more stuff without increasing the process VM
size any further.

Since the memory is no longer being actively used the VM system will slowly
reclaim it (by paging it out as it ages).

You can test this hypothesis by allocating big lumps of objects at the
Python prompt then del'ing them and watching the process memory stats. On my
machine, for instance, I get behaviour like so (with Python 2.3 from CVS on
Mac OS X):

>>> 
# RSIZE=1.63M, VSIZE=5.38M
>>> foo = [(x,y) for x in range(1000) for y in range(1000)]
# RSIZE=50.8M, VSIZE=58.5M
>>> del foo
# RSIZE=50.7M, VSIZE=54.7M
>>> bar = [(x,y) for x in range(1000) for y in range(1000)]
# RSIZE=50.8M, VSIZE=58.5M
>>> del bar
# RSIZE=50.7M, VSIZE=54.6M
>>> # leave Python in the background for a while doing nothing much
# RSIZE=692K, VSIZE=54.8M

which is pretty normal.

Jonathan