Detecting memory leaks on apache, mod_python

[Steven D'Aprano]

On Sat, 22 Dec 2007 08:09:50 +0100, Fredrik Lundh wrote:

> Steven D'Aprano wrote:
> 
>  > Not me.
> 
> You're quite knew to this internet thing, aren't you? ;-)

:-D


>> So... how do you measure memory usage in Python? Every programming
>> language I've used before (not a huge range, I'll admit) had *some*
>> sort of facility to measure memory usage, typically things like:
>> 
>> * how much memory is free in the stack?
>> 
>> * how much memory is free in the heap?
>> 
>> * how big a block does this pointer point to?
>> 
>> * how much memory does this record/struct/object/string use?
> 
> And what languages would that be?  I cannot think of a single modern
> language that does any of that.  Including low-level stuff like C/C++.

I didn't actually say they were *modern* languages. E.g. THINK Pascal for 
Apple Mac, circa 1990.


> And things like "how much memory is free in the heap" isn't even a
> meaningful concept on a modern machine, thanks to the wonders of virtual
> memory (especially the overcommitting kind).

Maybe the memory model used in modern multi-tasking virtual-memory PCs 
makes the concept of "free memory" obsolete. Although if so, somebody 
should have a quiet word with the author of the Linux free command:

$ free
             total       used       free     shared    buffers     cached
Mem:       1002524     988736      13788          0       7044      98916
-/+ buffers/cache:     882776     119748
Swap:      4241080    3939736     301344

(Admittedly that's system-wide memory usage, rather than for a single 
process.)



> For Python, standard process monitoring tools (combined with a basic
> understanding of how dynamic memory allocation works on modern
> platforms) are usually sufficient to get a good view of an application's
> memory usage patterns.  Just run the program under a few different
> scenarios, and see what it does.

Are you saying that Python programs can't monitor their own memory use?

I'm happy to accept that "free memory" is not a meaningful concept for a 
process in a modern system. That makes sense. But surely it is reasonable 
for a process to have an idea of how much memory it has actually used. 
Yes? No?


> If the memory use looks suspicious,
> use standard debugging techniques to locate the problematic area, and
> standard benchmarking techniques to look for unexpected blowups and
> leaks.

What sort of "standard debugging techniques" work in the absence of any 
way (that I know of) to measure memory usage? In Python, standard 
debugging techniques usually start with the print statement, but one 
can't do anything like this:

# problematic area of code
last = memory()
for i in xrange(100):
    x = foo()
    if memory() >= last:
        print "memory use increased", memory()


So what are you suggesting is standard?

 
> For really hard problems, use the "gc" module, instrumenting memory
> allocation libraries (e.g. dmalloc), or C/C++-level debuggers.

I'm not so much concerned about the really hard problems as I am about 
the really easy ones.



-- 
Steven