Hunting a memory leak

[Michael Hudson]

Debian User <falted at inspiron.openlc.org> writes:

> I'm trying to discover a memory leak on a program of mine. I've taken
> several approaches, but the leak still resists to appear.
> 
> First of all, I've tried to use the garbage collector to look for
> uncollectable objects.

[snip]

> Then I've taken an approach that I've seen in python developers list
> contributed by Walter Dörwald, that basically consists in creating a
> debug version of python, create a unitest with the leaking code, and
> modify the unittest.py to extract the increment of total reference
> counting in that code (see
> http://aspn.activestate.com/ASPN/Mail/Message/python-dev/1770868).

Well, somewhere in that same thread are various references to a
TrackRefs class.  Have you tried using that?  It should tell you what
type of object is leaking, which is a good start.

> With that, I see that my reference count grows by one each time the
> test execute. But the problem is: is there some way to look at the
> object (or make a memory dump) that is leaking?.

See above :-)

> I've used valgrind (http://developer.kde.org/~sewardj/) to see if it
> could detect the leak. In fact, it detects a bunch of them, but I am
> afraid that they are not related with the leak I'm looking for. I am
> saying that because, when I loop over my leaky code, valgrind always
> report the same amount of leaky memory, independently of the number of
> iterations (while top is telling me that memory use is growing!).

There are various things (interned strings, f'ex) that always tend to
be alive at the end of a Python program: these are only leaks in a
very warped sense.

I don't know if there's a way to get vaglrind to tell you what's
allocated but not deallocated between two arbitrary points of program
execution.

> My code uses extension modules in C, so I am afraid this does not
> contribute to alleviate the problem.

Well, in all likelyhood, the bug is IN the C extension module.  Have
you tried stepping through the code in a debugger?  Sometime's that's
a good way of spotting a logic error.

> I am sorry, but I cannot be more explicit about the code because it
> is quite complex (it is the PyTables package, http://pytables.sf.net),
> and I was unable to make a simple example to be published
> here. However, if anyone is tempted to have a look at the code, you
> can download it from
> (http://sourceforge.net/project/showfiles.php?group_id=63486). I am
> attaching a unittest that exposes the leak.
> 
> I am a bit desperate. Any hint?

Not really.  Try using TrackRefs.

Cheers,
mwh

-- 
  I'm about to search Google for contract assassins to go to Iomega
  and HP's programming groups and kill everyone there with some kind
  of electrically charged rusty barbed thing.
                -- http://bofhcam.org/journal/journal.html, 2002-01-08