How to optimize and monitor garbage collection?

[Terry Reedy]

On 10/24/2010 8:39 PM, kj wrote:

> I'm designing a system that will be very memory hungry unless it
> is "garbage-collected" very aggressively.
>
> In the past I have had disappointing results with the gc module:
> I noticed practically no difference in memory usage with and without
> it.  It is possible, however, that I was not measuring memory
> consumption adequately.
>
> What's the most accurate way to monitor memory consumption in a
> Python program, and thereby ensure that gc is working properly?
>
> Also, are there programming techniques that will result in better
> garbage collection?  For example, would it help to explicitly call
> del on objects that should be gc'd?

Python the language is not much concerned with memory. For an 
interpreter running on a computer, there are four memory sizes to be 
considered: the virtual memory assigned to the process; the physical 
memory assigned to the process; the physical memory used by Python 
objects; and the the memory used by 'active' objects accessible from 
program code. As far as I know, the OS can only see and report on the 
first and/or second.

If the gap between the second and third (assigned and used physical 
memory) is large and includes 'blocks' that are totally unused, the 
interpreter *may* be able to return such blocks. But do not count on it. 
When this gap expands because the program deletes objects without 
returning blocks, people get fooled by OS reports of assigned memory not 
shrinking (even though used memory is).

CPython tries to minimize the gap between all objects and active object 
both with reference counting and cyclic garbage collection (gc). Yes, 
you can help this along by judicious use of del and gc.collect. The goal 
should be to minimize the maximum active memory size. Reusing large 
arrays (rather than deleting and creating a new one) can sometimes help 
by avoiding fragmentation of allocated memory.

Terry Jan Reedy