Command line: (longexp crashes purify for some reason) ./python -E -tt ./Lib/test/regrtest.py -u network -x test_longexp.py Results: 168 tests OK. 1 test failed: test_socket_ssl 15 tests skipped: test_al test_bsddb test_cd test_cl test_curses test_dl test_gl test_imgfile test_linuxaudiodev test_nis test_ntpath test_openpty test_sundry test_winreg test_winsound test test_socket_ssl crashed -- socket.sslerror: SSL_CTX_new error When running test_curses: test test_curses crashed -- _curses.error: curs_set() returned ERR Uninitialized Memory Reads (don't think these are python's fault): gdbm_open() gdbmmodule.c:68 getaddrinfo() socketmodule.c:636 Memory Leaks: NONE Potential Memory Leaks: 1012 from 4 different allocating points -- all in WeakRef Things look great! Let me know if any of the skipped tests ought to have been run. Most likely possibility seems to be bsddb, sundry also needs bsddb. Neal
A few of these skips weren't expected: test_sundry, test_openpty, test_bsddb. What's missing to run those?
test test_socket_ssl crashed -- socket.sslerror: SSL_CTX_new error
Works for me. Probably your platform. The SSL code sucks anyway.
When running test_curses:
test test_curses crashed -- _curses.error: curs_set() returned ERR
Works for me.
Woohoo! So I'll close the remaining leak report!
Potential Memory Leaks:
1012 from 4 different allocating points -- all in WeakRef
Any more info on these, or are these just red herrings?
Things look great!
Agreed.
Let me know if any of the skipped tests ought to have been run. Most likely possibility seems to be bsddb, sundry also needs bsddb.
Ah, that's why sundry fails. I think it shouldn't depend on bsddb; I'll look into that. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
The platform is Sun Solaris 8 (2.8). Purify 2002. For bsddb, do I need to install something that isn't "standard"? test test_openpty skipped -- No openpty() available.
The details are below. Neal -- PLK: 704 bytes potentially leaked in 16 blocks This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] weakref_ref [_weakref.c:73] PyCFunction_Call [methodobject.c:101] eval_frame [ceval.c:1989] PyEval_EvalCodeEx [ceval.c:2570] function_call [funcobject.c:374] PyObject_Call [abstract.c:1665] instancemethod_call [classobject.c:2276] PyObject_Call [abstract.c:1665] PyEval_CallObjectWithKeywords [ceval.c:3034] Block of 44 bytes (16 times); last block at 0x18bf0a8 -- PLK: 220 bytes potentially leaked in 5 blocks This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] type_call [typeobject.c:148] PyObject_Call [abstract.c:1665] PyEval_CallObjectWithKeywords [ceval.c:3034] PyObject_CallFunction [abstract.c:1656] build_class [ceval.c:3583] eval_frame [ceval.c:1507] Block of 44 bytes (5 times); last block at 0xced9b8 -- PLK: 44 bytes potentially leaked at 0xcb2ba8 This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] tp_new_wrapper [typeobject.c:2586] PyCFunction_Call [methodobject.c:69] PyObject_Call [abstract.c:1665] do_call [ceval.c:3247] eval_frame [ceval.c:1984] PyEval_EvalCodeEx [ceval.c:2570] -- PLK: 44 bytes potentially leaked at 0xcb3470 This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] tp_new_wrapper [typeobject.c:2586] PyCFunction_Call [methodobject.c:69] PyObject_Call [abstract.c:1665] slot_tp_new [typeobject.c:3355] type_call [typeobject.c:148] PyObject_Call [abstract.c:1665]
Neal Norwitz's Purify writes:
Free'd weakref objects are stored in a free list, so it makes sense that these are labelled potential leaks. We could either give up using a free list for these, or we could allocate blocks of these rather than allocating them individually. The latter would allow still better performance and would reduce the malloc overhead. That change would be a higher risk than tossing the free list, and tossing it would be a higher risk than keeping it this close to the release. But we're probably far enough away that any of the options (no free list, blocking allocation, and leaving it alone) are manageable. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
Note that there are tons of memory that is validly still alive when Python exists. As we've seen, trying to clean all that up /can/ have a big negative impact on shutdown. So I wouldn't worry about them. Insure labels them as (paraphrasing) "memory still allocated at exit", distinct from memory leaked. I think this is equivalent to Purify's "potentially leaked" and "leaked" memory respectively. Note that at least stringobject.c has a hook for clearing out all those interned strings at program shutdown time, which I added to clean up Insure's output. When INTERN_STRINGS is defined, the extra function _Py_ReleaseInternedStrings() is defined. Then, in Modules/main.c, in Py_Main(), if __INSURE__ is defined that function is called, so this memory doesn't show up in the report. It may be worthwhile generalizing this approach, and adding it to other static long-lived data structures, simply as an aid to memory debugging. As long as it isn't enabled by default, that should be fine. -Barry
I don't think so. Insure (uselessly) lists *all* memory that's still allocated and reachable at exit. Purify only reports certain blocks (in this case they were a bunch of weak refs) as potentially leaked. I'm not sure what makes them potentially leaked, but it must be a stronger condition than "still exists at exit". As Fred mentioned, this may have to do with the weaklist free list implementation.
Most types that have a custom free list already have a cleanup routine that's always called by Py_Finalize(); the weakref free list is lacking one. The interned list is special because it's generally unsafe to clear it out when Py_Finalize() might be followed by Py_Initialize() again -- that's why it's only done when running Insure++. --Guido van Rossum (home page: http://www.python.org/~guido/)
guido wrote:
purify runs a garbage collection algorithm on your program to figure out if something has leaked. iirc, a potential memory leak is when you no longer have a pointer to the beginning of an allocated block, but there's a valid pointer to somewhere inside the block. a true leak is when you no longer point to the block (i.e when even a conservative garbage collector could safely destroy it). </F>
Thanks for the explanation! The simplest way to avoid the Purify "potential leak" complaints then might be to add a way to explicitly zap this free list at Py_Finalize() time, like we do with other custom free lists. That should be a post-2.2 feature. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Fred L. Drake, Jr.]
-0. When we turn on pymalloc in 2.3, blocks will get allocated by magic, and more space-efficiently than you're going to do by hand (it bites one malloc overhead per 256KB). pymalloc will likely be a little slower than a wholly customized free list, but it will get all the memory benefit, and unlike type-specific free lists allows memory to be recycled across types as programs progress.
There's another possibility: as float objects do, add a module xxx_fini() routine to clear the free list at shutdown.
A few of these skips weren't expected: test_sundry, test_openpty, test_bsddb. What's missing to run those?
test test_socket_ssl crashed -- socket.sslerror: SSL_CTX_new error
Works for me. Probably your platform. The SSL code sucks anyway.
When running test_curses:
test test_curses crashed -- _curses.error: curs_set() returned ERR
Works for me.
Woohoo! So I'll close the remaining leak report!
Potential Memory Leaks:
1012 from 4 different allocating points -- all in WeakRef
Any more info on these, or are these just red herrings?
Things look great!
Agreed.
Let me know if any of the skipped tests ought to have been run. Most likely possibility seems to be bsddb, sundry also needs bsddb.
Ah, that's why sundry fails. I think it shouldn't depend on bsddb; I'll look into that. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
The platform is Sun Solaris 8 (2.8). Purify 2002. For bsddb, do I need to install something that isn't "standard"? test test_openpty skipped -- No openpty() available.
The details are below. Neal -- PLK: 704 bytes potentially leaked in 16 blocks This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] weakref_ref [_weakref.c:73] PyCFunction_Call [methodobject.c:101] eval_frame [ceval.c:1989] PyEval_EvalCodeEx [ceval.c:2570] function_call [funcobject.c:374] PyObject_Call [abstract.c:1665] instancemethod_call [classobject.c:2276] PyObject_Call [abstract.c:1665] PyEval_CallObjectWithKeywords [ceval.c:3034] Block of 44 bytes (16 times); last block at 0x18bf0a8 -- PLK: 220 bytes potentially leaked in 5 blocks This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] type_call [typeobject.c:148] PyObject_Call [abstract.c:1665] PyEval_CallObjectWithKeywords [ceval.c:3034] PyObject_CallFunction [abstract.c:1656] build_class [ceval.c:3583] eval_frame [ceval.c:1507] Block of 44 bytes (5 times); last block at 0xced9b8 -- PLK: 44 bytes potentially leaked at 0xcb2ba8 This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] tp_new_wrapper [typeobject.c:2586] PyCFunction_Call [methodobject.c:69] PyObject_Call [abstract.c:1665] do_call [ceval.c:3247] eval_frame [ceval.c:1984] PyEval_EvalCodeEx [ceval.c:2570] -- PLK: 44 bytes potentially leaked at 0xcb3470 This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] tp_new_wrapper [typeobject.c:2586] PyCFunction_Call [methodobject.c:69] PyObject_Call [abstract.c:1665] slot_tp_new [typeobject.c:3355] type_call [typeobject.c:148] PyObject_Call [abstract.c:1665]
Neal Norwitz's Purify writes:
Free'd weakref objects are stored in a free list, so it makes sense that these are labelled potential leaks. We could either give up using a free list for these, or we could allocate blocks of these rather than allocating them individually. The latter would allow still better performance and would reduce the malloc overhead. That change would be a higher risk than tossing the free list, and tossing it would be a higher risk than keeping it this close to the release. But we're probably far enough away that any of the options (no free list, blocking allocation, and leaving it alone) are manageable. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
Note that there are tons of memory that is validly still alive when Python exists. As we've seen, trying to clean all that up /can/ have a big negative impact on shutdown. So I wouldn't worry about them. Insure labels them as (paraphrasing) "memory still allocated at exit", distinct from memory leaked. I think this is equivalent to Purify's "potentially leaked" and "leaked" memory respectively. Note that at least stringobject.c has a hook for clearing out all those interned strings at program shutdown time, which I added to clean up Insure's output. When INTERN_STRINGS is defined, the extra function _Py_ReleaseInternedStrings() is defined. Then, in Modules/main.c, in Py_Main(), if __INSURE__ is defined that function is called, so this memory doesn't show up in the report. It may be worthwhile generalizing this approach, and adding it to other static long-lived data structures, simply as an aid to memory debugging. As long as it isn't enabled by default, that should be fine. -Barry
I don't think so. Insure (uselessly) lists *all* memory that's still allocated and reachable at exit. Purify only reports certain blocks (in this case they were a bunch of weak refs) as potentially leaked. I'm not sure what makes them potentially leaked, but it must be a stronger condition than "still exists at exit". As Fred mentioned, this may have to do with the weaklist free list implementation.
Most types that have a custom free list already have a cleanup routine that's always called by Py_Finalize(); the weakref free list is lacking one. The interned list is special because it's generally unsafe to clear it out when Py_Finalize() might be followed by Py_Initialize() again -- that's why it's only done when running Insure++. --Guido van Rossum (home page: http://www.python.org/~guido/)
guido wrote:
purify runs a garbage collection algorithm on your program to figure out if something has leaked. iirc, a potential memory leak is when you no longer have a pointer to the beginning of an allocated block, but there's a valid pointer to somewhere inside the block. a true leak is when you no longer point to the block (i.e when even a conservative garbage collector could safely destroy it). </F>
Thanks for the explanation! The simplest way to avoid the Purify "potential leak" complaints then might be to add a way to explicitly zap this free list at Py_Finalize() time, like we do with other custom free lists. That should be a post-2.2 feature. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Fred L. Drake, Jr.]
-0. When we turn on pymalloc in 2.3, blocks will get allocated by magic, and more space-efficiently than you're going to do by hand (it bites one malloc overhead per 256KB). pymalloc will likely be a little slower than a wholly customized free list, but it will get all the memory benefit, and unlike type-specific free lists allows memory to be recycled across types as programs progress.
There's another possibility: as float objects do, add a module xxx_fini() routine to clear the free list at shutdown.
participants (6)
-
barry＠zope.com -
Fred L. Drake, Jr. -
Fredrik Lundh -
Guido van Rossum -
Neal Norwitz -
Tim Peters