[Baypiggies] find memory leaks in running program

Shannon -jj Behrens jjinux at gmail.com
Wed Dec 8 20:29:23 CET 2010

On Tue, Dec 7, 2010 at 9:04 AM, Marco Hornung <pythonmarco at gmail.com> wrote:
> Hey,
> ------------------------------------------------------------------------------------------
> questions
> ------------------------------------------------------------------------------------------
> 1. What are the best tools to analyze pythons memory stack, while it is running?
> 2. Is there a possibility to analyze the memory stack of a program with external programs? (without to change the source code - I am only interested in the object size)
> 3. Can I sort of "break" into the memory to see what objects consume how much memory?
> ------------------------------------------------------------------------------------------
> my scenario
> ------------------------------------------------------------------------------------------
> I have to debug a multithreaded server, which is written in the twisted-framework. One of the processes has some sort of memory leak - After one of our jobs is finished the main process has still over 59% of the entire memory allocated. I will probably have to recreate our scenario and equip our server with some memory sensors - but  it takes 12h to reproduce the scenario and I will have to change the source code(at least I do not know of other options).
> Therefore I am looking for quicker possibilities to look into what causes our memory leak.

One time I found a memory leak in MySQLdb.  It was in C code, and it
only happened when you passed a certain argument to the connection
parameters that had something to do with decoding.  It would only
manifest itself after a day.  That was really, really hard to find
because the leak was invisible from Python's perspective.  My approach
was to use a process of elimination.  I would slowly remove more and
more code and try to run the code faster and faster in a loop (making
external requests to my server) in order to detect the memory leak
more quickly.  It took a few days of effort.

By the way, it might not be completely relevant, but there was a
*fantastic* talk on debugging Ruby on Rails applications using
low-level tools:


A lot of it also applies to Python, but not all of it.  That talks was
simply incredible.  The speaker had used his techniques (which
involved a huge range of tools) to find multiple memory leaks and
massive performance problems in both Ruby and Rails.

Best Regards,

In this life we cannot do great things. We can only do small things
with great love. -- Mother Teresa

More information about the Baypiggies mailing list