help for mem leak debugging

[Will Ware]

Numerous postings to c.l.python, including some of my own, have
mentioned the difficulty of debugging memory leaks in C extensions. To
old hands this stuff may all be obvious, but we less experienced folk
need all the help we can get. This is a helpful trick I've developed
using the 1.5.2 codebase. I imagine it would work with the 1.6 and 2.0
codebases as well.

The typical problem with memory leaks is mismanagement of reference
counts, particularly abuses of Py_INCREF and Py_DECREF, as well as
ignorance of the refcount effects of functions like Py_BuildValue,
PyArg_ParseTuple, PyTuple/List_SetItem/GetItem, and so forth. The
existing codebase offers some help with this (search for
Py_TRACE_REFS) but I found it useful to add this function in
Objects/object.c, just before _Py_PrintReferences.

void
_Py_CountReferences(fp)
	FILE *fp;
{
	int n;
	PyObject *op;
	for (n = 0, op = refchain._ob_next;
	     op != &refchain;
	     op = op->_ob_next, /*n++*/ n += op->ob_refcnt);
        fprintf(fp, "%d refs\n", n);
}

The difference between this and _Py_PrintReferences is that the
latter prints out all objects and their refcounts, in a list that
runs many pages. It's obviously impractical to do that at several
points in a program that runs in a long loop. But this function
will only print the total of all the refcounts in the system, which
will allow you to keep track of when you have inadvertently put in
something that increases the total refcount every time going thru a
loop.

In my C extension, I put in the following macros.

#if defined(Py_DEBUG) || defined(DEBUG)
extern void _Py_CountReferences(FILE*);
#define CURIOUS(x) { fprintf(stderr, __FILE__ ":%d ", __LINE__); x; }
#else
#define CURIOUS(x)
#endif
#define MARKER()        CURIOUS(fprintf(stderr, "\n"))
#define DESCRIBE(x)     CURIOUS(fprintf(stderr, "  " #x "=%d\n", x))
#define DESCRIBE_HEX(x) CURIOUS(fprintf(stderr, "  " #x "=%08x\n", x))
#define COUNTREFS()     CURIOUS(_Py_CountReferences(stderr))

To debug, I rebuild Python using 'make OPT="-DPy_DEBUG"', which
causes the stuff under Py_TRACE_REFS to be built. My own makefile
uses the same trick, by including these lines:

debug:
        make clean; make OPT="-g -DPy_DEBUG" all
CFLAGS = $(OPT) -fpic -O2 -I/usr/local/include -I/usr/include/python1.5

When I suspect that one of my functions is responsible for a memory
leak, I liberally sprinkle it with calls to the COUNTREFS() macro.
This allows me to keep track of exactly how many references are being
created or destroyed as I go through my function. This is particularly
useful in loops where simple mistakes can cause reference counts to
grow ridiculously fast. Also, reference counts that shrink too fast
(overzealous use of Py_DECREF) can cause core dumps because the memory
for objects that should still exist has been reallocated for new
objects.

This might be a handy thing to throw in the FAQTS database.

-- 
# - - - - - - - - - - - - - - - - - - - - - - - -
# Resistance is futile. Capacitance is efficacious.
# Will Ware	email:    wware @ world.std.com