I checked in several changes to support valgrind better. Mostly it was documentation changes (additions). The changes should be minimal enough to be considered for backporting to 2.3.5. Although I did not mark all the checkins as such. At the end of this mail are the remaining issues that valgrind reports. There were very few problems reported. Valgrind was run on gentoo, dual amd athlon box with 1GB. I checked in a fix to Modules/binascii.c. Someone familiar with the module should review this change. It fixed an uninitialized memory read. It would be nice to improve the comment. I fixed a memory leak in Modules/posixmodule.c when callin utime(). The code is a bit complex, so it would be good to review this change carefully as well. Neal -- These are from most important to fix to least important. I submitted a patch to fix a bunch of possible memory leaks. the patch fixes at least one memory leak not reported here. The fixes are in the same location as these leaks. http://sourceforge.net/tracker/index.php?func=detail&aid=967763&group_id=5470&atid=305470 BSD DB memory leaks: -------------------- 4 bytes in 1 blocks are definitely lost in loss record 7 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C928265: make_key_dbt (_bsddb.c:400) by 0x3C92DE4F: DBC_set_range (_bsddb.c:2911) 8 bytes in 2 blocks are definitely lost in loss record 29 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C928265: make_key_dbt (_bsddb.c:400) by 0x3C929088: DB_get (_bsddb.c:1349) I'm not sure which test caused these leaks. I will try narrow these two down. I suspect one may be test_import.py. Import/compile related memory leaks: ------------------------------------ 128 bytes in 1 blocks are definitely lost in loss record 221 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x807CDB7: new_arena (obmalloc.c:500) by 0x807CAF2: PyObject_Malloc (obmalloc.c:699) by 0x807D5A7: PyString_FromStringAndSize (stringobject.c:73) by 0x80D6D43: r_object (marshal.c:500) by 0x80D63D5: r_object (marshal.c:621) by 0x80D5E01: r_object (marshal.c:546) by 0x80D6380: r_object (marshal.c:616) by 0x80D7683: PyMarshal_ReadLastObjectFromFile (marshal.c:665) by 0x80D0CAA: load_source_module (import.c:690) by 0x80CEA8D: load_module (import.c:1616) by 0x80CF977: import_submodule (import.c:2210) by 0x80CF4A2: load_next (import.c:2030) by 0x80D14D8: import_module_ex (import.c:1865) by 0x80D054B: PyImport_ImportModuleEx (import.c:1906) 256 bytes in 1 blocks are definitely lost in loss record 252 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x807CDB7: new_arena (obmalloc.c:500) by 0x807CAF2: PyObject_Malloc (obmalloc.c:699) by 0x80F8914: PyNode_AddChild (node.c:95) by 0x80F8BE1: PyParser_AddToken (parser.c:112) by 0x8055AF2: parsetok (parsetok.c:166) by 0x80DBFAD: Py_CompileStringFlags (pythonrun.c:1375) by 0x80A2AAC: builtin_compile (bltinmodule.c:391) by 0x81019F9: PyCFunction_Call (methodobject.c:108) by 0x80AA71D: call_function (ceval.c:3377) by 0x80AE5FF: eval_frame (ceval.c:1990) by 0x80A93AD: PyEval_EvalCodeEx (ceval.c:2543) by 0x80AA898: fast_function (ceval.c:3468) by 0x80AA3F3: call_function (ceval.c:3397) by 0x80AE5FF: eval_frame (ceval.c:1990) The thread leaks may occur at most once in a program. I'm not sure. Thread memory leaks: -------------------- 16 bytes in 1 blocks are definitely lost in loss record 63 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x80DF227: PyThread_allocate_lock (thread_pthread.h:262) by 0x80ABC43: PyEval_InitThreads (ceval.c:162) by 0x80E287F: thread_PyThread_start_new_thread (threadmodule.c:243) 64 bytes in 4 blocks are definitely lost in loss record 132 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x80E2843: thread_PyThread_start_new_thread (threadmodule.c:233) Initialization memory leaks: ---------------------------- 64 bytes in 1 blocks are definitely lost in loss record 139 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x807CE32: new_arena (obmalloc.c:452) by 0x807CAF2: PyObject_Malloc (obmalloc.c:699) by 0x80E1566: _PyObject_GC_Malloc (gcmodule.c:1183) by 0x80E166C: _PyObject_GC_NewVar (gcmodule.c:1214) by 0x8084A71: PyTuple_New (tupleobject.c:68) by 0x8087B2F: PyType_Ready (typeobject.c:3161) by 0x8087B39: PyType_Ready (typeobject.c:3147) by 0x807BE01: _Py_ReadyTypes (object.c:1802) by 0x80D9B93: Py_Initialize (pythonrun.c:167) by 0x805510A: Py_Main (main.c:376) by 0x8054DAA: main (python.c:23) I don't know if we can fix any of these. They may all be within non-Python libraries. DL memory leaks: ---------------- 17 bytes in 1 blocks are definitely lost in loss record 90 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C0060D9: _dl_map_object (in /lib/ld-2.3.2.so) by 0x3C25E0AA: (within /lib/libc-2.3.2.so) by 0x3C00B247: _dl_catch_error (in /lib/ld-2.3.2.so) by 0x3C25E6C8: _dl_open (in /lib/libc-2.3.2.so) by 0x3C06DCF7: (within /lib/libdl-2.3.2.so) by 0x3C00B247: _dl_catch_error (in /lib/ld-2.3.2.so) by 0x3C06D485: (within /lib/libdl-2.3.2.so) by 0x3C06DD4B: dlopen (in /lib/libdl-2.3.2.so) by 0x3F9DD3E2: dl_open (dlmodule.c:184) Readline memory leaks: ---------------------- 40 bytes in 2 blocks are definitely lost in loss record 116 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3CB85EF1: xmalloc (in /lib/libreadline.so.4.3) Reading password/group memory leaks: ------------------------------------ 36 bytes in 1 blocks are definitely lost in loss record 162 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C23EBC2: (within /lib/libc-2.3.2.so) by 0x3C23E4F2: __nss_database_lookup (in /lib/libc-2.3.2.so) by 0x3FA3533D: ??? by 0x3FA355B0: ??? by 0x3C23F5FD: (within /lib/libc-2.3.2.so) by 0x3C200131: setgrent (in /lib/libc-2.3.2.so) by 0x3CA32E17: grp_getgrall (grpmodule.c:123) 36 bytes in 1 blocks are definitely lost in loss record 163 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C23EBC2: (within /lib/libc-2.3.2.so) by 0x3C23E4F2: __nss_database_lookup (in /lib/libc-2.3.2.so) by 0x3FA362FD: ??? by 0x3FA3780C: ??? by 0x3C201597: getpwnam_r (in /lib/libc-2.3.2.so) by 0x3C20103D: getpwnam (in /lib/libc-2.3.2.so) by 0x3CB48D80: pwd_getpwnam (pwdmodule.c:130)
[Neal Norwitz] ...
Import/compile related memory leaks: ------------------------------------
These aren't actually related to anything <wink>. The code starting at
by 0x807CDB7: new_arena (obmalloc.c:500)
is this: p = (uptr *)malloc(newmax * sizeof(*arenas)); if (p == NULL) goto error; memcpy(p, arenas, narenas * sizeof(*arenas)); arenas = p; /* old arenas deliberately leaked */ Note the comment on the last line! The reason for this is explained in horrid detail in a comment block a few lines above this. pymalloc maintains a vector of pointers to arena base addresses, and when this vector itself needs to grow, the memory it used to occupy is allowed to leak. In the absence of introducing expensive thread locks, we can't know whether some other thread may still be referencing the old version of this vector (it's OK if it does), so we can't free() it. Any valgrind complaint with
by 0x807CDB7: new_arena (obmalloc.c:500)
in the listing is due to this deliberate leak, and the code lower in the listing is irrelevant. It just had the bad luck to call pymalloc exactly when available space in all currently allocated arenas is in use, *and* there's no space left in pymalloc's exponentially-overallocated vector of arena base addresses to hold one more arena base address. ...
Initialization memory leaks: ----------------------------
64 bytes in 1 blocks are definitely lost in loss record 139 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x807CE32: new_arena (obmalloc.c:452) by 0x807CAF2: PyObject_Malloc (obmalloc.c:699) by 0x80E1566: _PyObject_GC_Malloc (gcmodule.c:1183) by 0x80E166C: _PyObject_GC_NewVar (gcmodule.c:1214) by 0x8084A71: PyTuple_New (tupleobject.c:68) by 0x8087B2F: PyType_Ready (typeobject.c:3161) by 0x8087B39: PyType_Ready (typeobject.c:3147) by 0x807BE01: _Py_ReadyTypes (object.c:1802) by 0x80D9B93: Py_Initialize (pythonrun.c:167) by 0x805510A: Py_Main (main.c:376) by 0x8054DAA: main (python.c:23)
That's really the same thing: this is the *first* time the vector of arena base addresses got allocated. That memory is eventually allowed to leak by the C code reproduced above. This isn't worth any convolution "to fix". Each pointer we save in the vector of arena base addresses controls 256KB of memory, so it's an insignificant space overhead. The vector doubles in size whenever growth is necessary, so the total amount of leaked space is approximately equal to the current size of the vector.
participants (2)
-
Neal Norwitz
-
Tim Peters