Py_Finalize does not release all memory, not even closely
Several places in the documentation claim that Py_Finalize will release all memory: http://docs.python.org/api/embedding.html says that literally, and http://docs.python.org/api/initialization.html#l2h-778 suggests it is a bug when memory is not freed in Py_Finalize. This has left people to believe that this is a bug: https://sourceforge.net/tracker/index.php?func=detail&aid=1445210&group_id=5470&atid=105470 However, I don't see any chance to make this promise even remotely. Objects allocated in extension modules, and held in global variables (e.g. socketmodule.c:socket_error, socket_herror, socket_gaierror, socket_timeout) will never be released, right? And because of the small objects allocator, their pool will remain allocated, right? And, then, the arena. So ISTM that invoking Py_Finalize after importing socket will yield atleast 256KiB garbage. Of course, that's not real garbage, because the next Py_Initialize'd interpreter will continue to allocate from the arenas. But still, the actual objects that the modules hold on to will not be reclaimed until the process terminates. Please correct me if I'm wrong. Regards, Martin
I'm afraid it was wishful thinking on my part.
The best we can try to hope for is to ensure that repeatedly calling
Py_Initialize and Py_Finalize doesn't leak too much memory.
--Guido
On 4/11/06, "Martin v. Löwis"
Several places in the documentation claim that Py_Finalize will release all memory:
http://docs.python.org/api/embedding.html
says that literally, and
http://docs.python.org/api/initialization.html#l2h-778
suggests it is a bug when memory is not freed in Py_Finalize.
This has left people to believe that this is a bug:
https://sourceforge.net/tracker/index.php?func=detail&aid=1445210&group_id=5470&atid=105470
However, I don't see any chance to make this promise even remotely. Objects allocated in extension modules, and held in global variables (e.g. socketmodule.c:socket_error, socket_herror, socket_gaierror, socket_timeout) will never be released, right?
And because of the small objects allocator, their pool will remain allocated, right? And, then, the arena.
So ISTM that invoking Py_Finalize after importing socket will yield atleast 256KiB garbage. Of course, that's not real garbage, because the next Py_Initialize'd interpreter will continue to allocate from the arenas. But still, the actual objects that the modules hold on to will not be reclaimed until the process terminates.
Please correct me if I'm wrong.
Regards, Martin
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I'm afraid it was wishful thinking on my part.
The best we can try to hope for is to ensure that repeatedly calling Py_Initialize and Py_Finalize doesn't leak too much memory.
Ok. Unless somebody still claims to the contrary, I will go ahead and weaken the promises in the documentation. Then it's not a bug anymore that it leaks :-) Regards, Martin
[Martin v. Löwis]
Several places in the documentation claim that Py_Finalize will release all memory:
http://docs.python.org/api/embedding.html
says that literally,
It's wrong ;-).
and
http://docs.python.org/api/initialization.html#l2h-778
suggests it is a bug when memory is not freed in Py_Finalize.
This has left people to believe that this is a bug:
https://sourceforge.net/tracker/index.php?func=detail&aid=1445210&group_id=5470&atid=105470
Well, there may well be a bug (or multiple bugs) underlying that one too. It's one thing for Py_Finalize() not to release all memory (it doesn't and probably never will), but it's not necessarily the same thing if running Py_Initialize() ... Py_Finalize() repeatedly keeps leaking more and more memory.
However, I don't see any chance to make this promise even remotely. Objects allocated in extension modules, and held in global variables (e.g. socketmodule.c:socket_error, socket_herror, socket_gaierror, socket_timeout) will never be released, right?
Not unless the module has a finalization function called by Py_Finalize() that frees such things (like PyString_Fini and PyInt_Fini). Other globals allocated via a static PyObject *someglobal = NULL; ... if (someglobal == NULL) someglobal = allocate_an_object_somehow(); pattern shouldn't contribute to continuing leaks across Py_Initialize() ... Py_Finalize() loops.
And because of the small objects allocator, their pool will remain allocated, right? And, then, the arena.
Before Python 2.5, arenas are never freed, period. In Python 2.5, an arena will be freed if and only if it contains no allocated object by the time Py_Finalize ends. There may also be trash cycles that aren't collected during Py_Finalize because I had to comment out Py_Finalize's second call to PyGC_Collect(); new-style class objects are among the trash thingies leaked (although if Py_Initialize() is called again, it's possible that they'll get cleaned up by cyclic gc after all). See Misc/SpecialBuilds.txt, section Py_TRACE_REFS, entry PYTHONDUMPREFS, for a way to get a dump of all heap objects Py_Finalize leaves alive. I doubt that's been run in years; Guido and I used it in 2.3b1 to cure some unreasonably high finalization leakage at the time.
So ISTM that invoking Py_Finalize after importing socket will yield atleast 256KiB garbage. Of course, that's not real garbage, because the next Py_Initialize'd interpreter will continue to allocate from the arenas.
I'm not clear on whether, e.g., init_socket() may get called more than once if socket-slinging code appears in a Py_Initialize() ... Py_Finalize(). If it doesn't, then, e.g., the unconditional socket_gaierror = PyErr_NewException("socket.gaierror", ...); won't contribute to ongoing leaks. But if it does, then we'll systematically leak the exception object on each loop trip.
But still, the actual objects that the modules hold on to will not be reclaimed until the process terminates.
Without a _Fini() function called by Py_Finalize, that's correct.
Please correct me if I'm wrong.
If you ever are, I will ;-)
Tim Peters wrote:
Well, there may well be a bug (or multiple bugs) underlying that one too. It's one thing for Py_Finalize() not to release all memory (it doesn't and probably never will), but it's not necessarily the same thing if running Py_Initialize() ... Py_Finalize() repeatedly keeps leaking more and more memory.
Running Py_Initialize/Py_Finalize once leaves 2150 objects behind (on Linux). The second run adds 180 additional objects; each subsequent run appears to add 156 more.
Not unless the module has a finalization function called by Py_Finalize() that frees such things (like PyString_Fini and PyInt_Fini).
How should the module install such a function? PyString_Fini and PyInt_Fini are invoked explicitly in pythonrun.c. That doesn't scale to extension modules.
I'm not clear on whether, e.g., init_socket() may get called more than once if socket-slinging code appears in a Py_Initialize() ... Py_Finalize().
Module initialization functions are called each time. Py_Finalize "forgets" which modules had been loaded, and reloads them all. Regards, Martin
Martin v. Löwis wrote:
Tim Peters wrote:
Well, there may well be a bug (or multiple bugs) underlying that one too. It's one thing for Py_Finalize() not to release all memory (it doesn't and probably never will), but it's not necessarily the same thing if running Py_Initialize() ... Py_Finalize() repeatedly keeps leaking more and more memory.
Running Py_Initialize/Py_Finalize once leaves 2150 objects behind (on Linux). The second run adds 180 additional objects; each subsequent run appears to add 156 more.
With COUNT_ALLOCS, I get the following results: Ignoring the two initial rounds of init/fini, each subsequent init/fini pair puts this number of objects into garbage: builtin_function_or_method 9 cell 1 code 12 dict 23 function 12 getset_descriptor 9 instancemethod 7 int 9 list 6 member_descriptor 23 method_descriptor 2 staticmethod 1 str 86 tuple 78 type 14 weakref 38 wrapper_descriptor 30 This totals to 360, which is for some reason higher than the numbers I get when counting the objects on the global list of objects. Is it not right to obtain the number of live object by computing tp->tp_allocs-tp->tp_frees? Regards, Martin
[Martin]
Running Py_Initialize/Py_Finalize once leaves 2150 objects behind (on Linux). The second run adds 180 additional objects; each subsequent run appears to add 156 more.
One thing I notice is that they're all > 0 :-) I believe that, at one time, the second and subsequent numbers were 0, but maybe that's just old age pining for the days when kids listened to good music. Because new-style classes create cycles that Py_Finalize() doesn't clean up, it may make analysis easier to stick a PyGC_Collect() call (or two! repeat until it returns 0) inside the loop now. ...
Not unless the module has a finalization function called by Py_Finalize() that frees such things (like PyString_Fini and PyInt_Fini).
How should the module install such a function?
There is no way at present, short of editing the source for Py_Finalize and recompiling. Presumably this is something that should be addressed in the module initialization/finalization PEP, right? I suppose people may want a way for Python modules to provide finalization functions too.
I'm not clear on whether, e.g., init_socket() may get called more than once if socket-slinging code appears in a Py_Initialize() ... Py_Finalize().
Module initialization functions are called each time. Py_Finalize "forgets" which modules had been loaded, and reloads them all.
OK, so things like the previous example (socketmodule.c's unconditional socket_gaierror = PyErr_NewException(...); in init_socket()) are guaranteed to leak "the old" socket_gaierror object across multiple socket module initializations. ... [later msg] ...
With COUNT_ALLOCS, I get the following results: Ignoring the two initial rounds of init/fini, each subsequent init/fini pair puts this number of objects into garbage:
builtin_function_or_method 9 cell 1 code 12 dict 23 function 12 getset_descriptor 9 instancemethod 7 int 9 list 6 member_descriptor 23 method_descriptor 2 staticmethod 1 str 86 tuple 78 type 14 weakref 38 wrapper_descriptor 30
FYI, from debugging Zope C-code leaks, staring at leftover strings, dict keys, and tuple contents often helps identify the sources. types too.
This totals to 360, which is for some reason higher than the numbers I get when counting the objects on the global list of objects.
How much higher? Last time I looked at this stuff (2.3b1, I think), the "all" in "the global list of all objects" wasn't true, and I made many changes to the core at the time to make it "more true". For example, all static C singleton objects were missing from that list (from Py_None and Py_True to all builtin static type objects). As SpecialBuilds.txt says, it became true in 2.3 that "a static type object T does appear in this list if at least one object of type T has been created" when COUNT_ALLOCS is defined. I expect that for _most_ static type objects, just doing initialize/finalize will not create objects of that type, so they'll be missing from the global list of objects. It used to be a good check to sum ob_refcnt over all objects in the global list, and compare that to _Py_RefTotal. The difference was a good clue about how many objects were missing from the global list, and became a better clue after I added code to inc_count() to call _Py_AddToAllObjects() whenever an incoming type object has tp_allocs == 0. Because of the latter, every type object with a non-zero tp_allocs count is in the list of all objects now (but only when COUNT_ALLOCS is defined). It's possible that other static singletons (like Py_None) have been defined since then that didn't grow code to add them to the global list, although by construction _PyBuiltin_Init() does add all available from __builtin__.
Is it not right to obtain the number of live object by computing tp->tp_allocs-tp->tp_frees?
That's the theory ;-) This code is never tested, though, and I bet is rarely used. Every time I've looked at it (most recently for 2.3b1, about 3 years ago), it was easy to find bugs. They creep in over time. Common historical weak points are "fancy" object-creation code, and the possibility of resurrection in a destructor. The special builds need special stuff at such times, and exactly what's needed isn't obvious.
Tim Peters wrote:
Because new-style classes create cycles that Py_Finalize() doesn't clean up, it may make analysis easier to stick a PyGC_Collect() call (or two! repeat until it returns 0) inside the loop now.
I'm shy to do this: the comment in Py_Finalize suggests that things will break if there is a "late" garbage collection.
There is no way at present, short of editing the source for Py_Finalize and recompiling. Presumably this is something that should be addressed in the module initialization/finalization PEP, right?
Indeed.
This totals to 360, which is for some reason higher than the numbers I get when counting the objects on the global list of objects.
How much higher?
Well, I counted an increase of 156 objects on the "all objects" list, and an increase of 360 according to the COUNT_ALLOCS numbers. The first number was without COUNT_ALLOCS being defined, though. Anyway, thanks for your comments. I'll try to look at this from time to time, maybe I can resolve some of the leaks. Regards, Martin
[Tim]
Because new-style classes create cycles that Py_Finalize() doesn't clean up, it may make analysis easier to stick a PyGC_Collect() call (or two! repeat until it returns 0) inside the loop now.
[Martin]
I'm shy to do this: the comment in Py_Finalize suggests that things will break if there is a "late" garbage collection.
Putting a collection call inside an initialize/finalize loop isn't doing it late, it's doing it early. If we can't collect cyclic trash after Py_Initialize(), that would be a showstopper for apps embedding Python "in a loop"! There's either nothing to fear here, or Python has a very bad bug. Are you thinking of this comment?: /* Collect final garbage. This disposes of cycles created by * new-style class definitions, for example. * XXX This is disabled because it caused too many problems. If * XXX a __del__ or weakref callback triggers here, Python code has * XXX a hard time running, because even the sys module has been * XXX cleared out (sys.stdout is gone, sys.excepthook is gone, etc). * XXX One symptom is a sequence of information-free messages * XXX coming from threads (if a __del__ or callback is invoked, * XXX other threads can execute too, and any exception they encounter * XXX triggers a comedy of errors as subsystem after subsystem * XXX fails to find what it *expects* to find in sys to help report * XXX the exception and consequent unexpected failures). I've also * XXX seen segfaults then, after adding print statements to the * XXX Python code getting called. */ I wrote that, and think it's pretty clear: after PyImport_Cleanup(), so little of the interpreter still exists that _any_ problem while running Python code has a way of turning into a fatal problem. For example, internal error mechanisms that fetch things from the sys module (like excepthook or stdout) are usually (always?) careful to check whether the fetched things are NULL, but wander into lala-land when Py_None comes back (as it does after PyImport_Cleanup() "Nones-out" everything in sys). But call Py_Initialize() again, and everything (including the sys module) should become usable again. ...
This totals to 360, which is for some reason higher than the numbers I get when counting the objects on the global list of objects.
How much higher?
Well, I counted an increase of 156 objects on the "all objects" list, and an increase of 360 according to the COUNT_ALLOCS numbers. The first number was without COUNT_ALLOCS being defined, though.
Anyway, thanks for your comments. I'll try to look at this from time to time, maybe I can resolve some of the leaks.
Could you check in the code you're using? It would be useful to share this, and improve it over time. Of course it's easy to write a C program that calls Py_Initialize() and Py_Finalize() in a loop, but it gets real tedious real fast to add analysis code sufficient to truly help track down leaks. I'd be happiest to contribute future hints in the form of working C code :-) Heck, if we got the leaks down to 0 on second and subsequent calls, we could even make a standard test out of it to ensure it stays that way.
Tim Peters wrote:
Putting a collection call inside an initialize/finalize loop isn't doing it late, it's doing it early. If we can't collect cyclic trash after Py_Initialize(), that would be a showstopper for apps embedding Python "in a loop"! There's either nothing to fear here, or Python has a very bad bug.
Right. I did that, and it collects 308 objects after the first call in the second "round" of Py_Initialize/Py_Finalize, and then no additional objects. However, I don't think that helps much: Py_Finalize will call PyGC_Collect(), anyway, and before any counts are made.
Are you thinking of this comment?:
Yes; I was assuming you suggested to enable that block of code.
I wrote that, and think it's pretty clear: after PyImport_Cleanup(), so little of the interpreter still exists that _any_ problem while running Python code has a way of turning into a fatal problem.
Right. I still haven't tried it, but it might be that, after a plain Py_Initialize/Py_Finalize sequence, no such problems will occur, and that it would be safe to call it in this specific case.
Could you check in the code you're using?
I had to modify code in ways that shouldn't be checked in, e.g. by putting API calls into _Py_PrintReferenceAddresses, even though the comment says it does't call any API. When I get to clean this up, I'll check it in. With some debugging, I now found a "leak" that contributes to quite some of these garbage objects: Each round of Py_Initialize/Py_Finalize will leave a CodecInfo type behind. I think it comes from this block of code /* Note that as of Python 2.2, heap-allocated type objects * can go away, but this code requires that they stay alive * until program exit. That's why we're careful with * refcounts here. type_list gets a new reference to tp, * while ownership of the reference type_list used to hold * (if any) was transferred to tp->tp_next in the line above. * tp is thus effectively immortal after this. */ Py_INCREF(tp); so that this "leak" would only exist if COUNT_ALLOCS is defined. I would guess that even more of the leaking type objects (16 per round) can be attributed to this. This completely obstructs measurements, and could well explain why the number of leaked objects is so much higher when COUNT_ALLOCS is defined. OTOH, I can see why "this code requires that they stay alive". Any ideas on how to solve this dilemma? Perhaps the type_list could be a list of weak references, so that the types do have a chance to go away when the last instance disappears? Regards, Martin
participants (3)
-
"Martin v. Löwis" -
Guido van Rossum -
Tim Peters