There are also a few fingers pointing in the direction of weakref_ref, e.g.

==28827== 520 bytes in 14 blocks are possibly lost in loss record 48 of 106 ==28827== at 0x400481B4: malloc (vg_clientfuncs.c:100) ==28827== by 0x8099519: _PyObject_GC_New (Modules/gcmodule.c:868) ==28827== by 0x8067BA5: PyWeakref_NewRef (Objects/weakrefobject.c:37) ==28827== by 0x8066119: add_subclass (Objects/typeobject.c:2249) ==28827== by 0x8061F29: PyType_Ready (Objects/typeobject.c:2219) ==28827== by 0x80605A8: type_new (Objects/typeobject.c:1280) ==28827== by 0x805EDA4: type_call (Objects/typeobject.c:183) ==28827== by 0x80ABB0C: PyObject_Call (Objects/abstract.c:1688) ==28827== by 0x807A34F: PyEval_CallObjectWithKeywords (Python/ceval.c:3058) ==28827== by 0x80AB0C6: PyObject_CallFunction (Objects/abstract.c:1679)

Of course many of these could be caused by a single leak that drops a pointer to a container -- then everything owned by that container is also leaked.

This one seems simple: diff -c -c -r2.126.4.25 typeobject.c *** typeobject.c 11 Oct 2002 00:22:22 -0000 2.126.4.25 --- typeobject.c 18 Oct 2002 00:50:31 -0000 *************** *** 2249,2256 **** while (--i >= 0) { ref = PyList_GET_ITEM(list, i); assert(PyWeakref_CheckRef(ref)); ! if (PyWeakref_GET_OBJECT(ref) == Py_None) ! return PyList_SetItem(list, i, new); } i = PyList_Append(list, new); Py_DECREF(new); --- 2249,2259 ---- while (--i >= 0) { ref = PyList_GET_ITEM(list, i); assert(PyWeakref_CheckRef(ref)); ! if (PyWeakref_GET_OBJECT(ref) == Py_None) { ! i = PyList_SetItem(list, i, new); ! Py_DECREF(new); ! return i; ! } } i = PyList_Append(list, new); Py_DECREF(new); --Guido van Rossum (home page: http://www.python.org/~guido/)

Just been lurkin' :) Possibly lost means that Valgrind didn't see any freeing for a specific point after a malloc. In valgrind terms, you probably forgot to free a pointer and lost all points that could reference that memory. In other words, a leak :) Usually 75% of those errors can be solved locally within the function, although I haven't looked at the specifics, but that 25% can get nasty. -- Mike On Thu, Oct 17 @ 20:38, Guido van Rossum wrote:

There are a few other records mentioning GC_Resize, but this one is the biggest. Could it be that the free frame list is botched? OTOH, what does "possibly lost" really mean?

-- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

This should help shed some light on the situation: Quoth the docs: """ For each such block, Valgrind scans the entire address space of the process, looking for pointers to the block. One of three situations may result: * A pointer to the start of the block is found. This usually indicates programming sloppiness; since the block is still pointed at, the programmer could, at least in principle, free'd it before program exit. * A pointer to the interior of the block is found. The pointer might originally have pointed to the start and have been moved along, or it might be entirely unrelated. Valgrind deems such a block as "dubious", that is, possibly leaked, because it's unclear whether or not a pointer to it still exists. * The worst outcome is that no pointer to the block can be found. The block is classified as "leaked", because the programmer could not possibly have free'd it at program exit, since no pointer to it exists. This might be a symptom of having lost the pointer at some earlier point in the program. """ Possibly is the second case and definitely lost is the third case. The definitely lost, in my experience, tends to mean you just forgot to free a pointer. The possibly lost usually means that some memory rot occurred, where it's not clear which pointer is causing the mem leak. -- Mike On Mon, Oct 28 @ 19:37, Guido van Rossum wrote:

...

Just been lurkin' :) Possibly lost means that Valgrind didn't see any freeing for a specific point after a malloc. In valgrind terms, you probably forgot to free a pointer and lost all points that could reference that memory. In other words, a leak :) Usually 75% of those errors can be solved locally within the function, although I haven't looked at the specifics, but that 25% can get nasty.

Booh.

That statistic doesn't apply to Python -- there are no "simple" leaks left in Python's C code, the remaining ones are all nasty.

Also, valgrind carefully distinguishes between "definitely lost" and "possibly lost". What's the difference?

--Guido van Rossum (home page: http://www.python.org/~guido/)

-- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

[Guido]

... (I wonder if pymalloc adds to the confusion, since its arenas count as a single block to malloc and hence to valgrind, but are internally cut up into many objects.)

For each arena, the address returned by malloc() is stored in the file-static arenas[] vector. So if a diagnostic program can find that vector, it can find the base address of every arena gotten from malloc. In a debug build, though, pymalloc adds pad bytes to both ends of each request (and whether handled by pymalloc or by malloc!), and returns the address of the byte beyond the leading pad byte. This can leave any number of system-malloc blocks with no direct pointer to their start. That's specific to the debug build, which forces all Python mem API calls to go thru pymalloc (the release build only directs PyObject_{Malloc,etc}() calls to pymalloc).

On Mon, Oct 28, 2002 at 08:58:45PM -0500, Michael Gilfix wrote:

I don't think valgrind reports this first case [a pointer to the beginning of the block is found]. Only the second and third, from what I gathered in the docs.

See --show-reachable

There are also a few fingers pointing in the direction of weakref_ref, e.g.

==28827== 520 bytes in 14 blocks are possibly lost in loss record 48 of 106 ==28827== at 0x400481B4: malloc (vg_clientfuncs.c:100) ==28827== by 0x8099519: _PyObject_GC_New (Modules/gcmodule.c:868) ==28827== by 0x8067BA5: PyWeakref_NewRef (Objects/weakrefobject.c:37) ==28827== by 0x8066119: add_subclass (Objects/typeobject.c:2249) ==28827== by 0x8061F29: PyType_Ready (Objects/typeobject.c:2219) ==28827== by 0x80605A8: type_new (Objects/typeobject.c:1280) ==28827== by 0x805EDA4: type_call (Objects/typeobject.c:183) ==28827== by 0x80ABB0C: PyObject_Call (Objects/abstract.c:1688) ==28827== by 0x807A34F: PyEval_CallObjectWithKeywords (Python/ceval.c:3058) ==28827== by 0x80AB0C6: PyObject_CallFunction (Objects/abstract.c:1679)

Of course many of these could be caused by a single leak that drops a pointer to a container -- then everything owned by that container is also leaked.

This one seems simple: diff -c -c -r2.126.4.25 typeobject.c *** typeobject.c 11 Oct 2002 00:22:22 -0000 2.126.4.25 --- typeobject.c 18 Oct 2002 00:50:31 -0000 *************** *** 2249,2256 **** while (--i >= 0) { ref = PyList_GET_ITEM(list, i); assert(PyWeakref_CheckRef(ref)); ! if (PyWeakref_GET_OBJECT(ref) == Py_None) ! return PyList_SetItem(list, i, new); } i = PyList_Append(list, new); Py_DECREF(new); --- 2249,2259 ---- while (--i >= 0) { ref = PyList_GET_ITEM(list, i); assert(PyWeakref_CheckRef(ref)); ! if (PyWeakref_GET_OBJECT(ref) == Py_None) { ! i = PyList_SetItem(list, i, new); ! Py_DECREF(new); ! return i; ! } } i = PyList_Append(list, new); Py_DECREF(new); --Guido van Rossum (home page: http://www.python.org/~guido/)

Just been lurkin' :) Possibly lost means that Valgrind didn't see any freeing for a specific point after a malloc. In valgrind terms, you probably forgot to free a pointer and lost all points that could reference that memory. In other words, a leak :) Usually 75% of those errors can be solved locally within the function, although I haven't looked at the specifics, but that 25% can get nasty. -- Mike On Thu, Oct 17 @ 20:38, Guido van Rossum wrote:

There are a few other records mentioning GC_Resize, but this one is the biggest. Could it be that the free frame list is botched? OTOH, what does "possibly lost" really mean?

-- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

This should help shed some light on the situation: Quoth the docs: """ For each such block, Valgrind scans the entire address space of the process, looking for pointers to the block. One of three situations may result: * A pointer to the start of the block is found. This usually indicates programming sloppiness; since the block is still pointed at, the programmer could, at least in principle, free'd it before program exit. * A pointer to the interior of the block is found. The pointer might originally have pointed to the start and have been moved along, or it might be entirely unrelated. Valgrind deems such a block as "dubious", that is, possibly leaked, because it's unclear whether or not a pointer to it still exists. * The worst outcome is that no pointer to the block can be found. The block is classified as "leaked", because the programmer could not possibly have free'd it at program exit, since no pointer to it exists. This might be a symptom of having lost the pointer at some earlier point in the program. """ Possibly is the second case and definitely lost is the third case. The definitely lost, in my experience, tends to mean you just forgot to free a pointer. The possibly lost usually means that some memory rot occurred, where it's not clear which pointer is causing the mem leak. -- Mike On Mon, Oct 28 @ 19:37, Guido van Rossum wrote:

...

Just been lurkin' :) Possibly lost means that Valgrind didn't see any freeing for a specific point after a malloc. In valgrind terms, you probably forgot to free a pointer and lost all points that could reference that memory. In other words, a leak :) Usually 75% of those errors can be solved locally within the function, although I haven't looked at the specifics, but that 25% can get nasty.

Booh.

That statistic doesn't apply to Python -- there are no "simple" leaks left in Python's C code, the remaining ones are all nasty.

Also, valgrind carefully distinguishes between "definitely lost" and "possibly lost". What's the difference?

--Guido van Rossum (home page: http://www.python.org/~guido/)

-- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

[Guido]

... (I wonder if pymalloc adds to the confusion, since its arenas count as a single block to malloc and hence to valgrind, but are internally cut up into many objects.)

For each arena, the address returned by malloc() is stored in the file-static arenas[] vector. So if a diagnostic program can find that vector, it can find the base address of every arena gotten from malloc. In a debug build, though, pymalloc adds pad bytes to both ends of each request (and whether handled by pymalloc or by malloc!), and returns the address of the byte beyond the leading pad byte. This can leave any number of system-malloc blocks with no direct pointer to their start. That's specific to the debug build, which forces all Python mem API calls to go thru pymalloc (the release build only directs PyObject_{Malloc,etc}() calls to pymalloc).

On Mon, Oct 28, 2002 at 08:58:45PM -0500, Michael Gilfix wrote:

I don't think valgrind reports this first case [a pointer to the beginning of the block is found]. Only the second and third, from what I gathered in the docs.

See --show-reachable