[Python-Dev] Py_Finalize does not release all memory, not even closely

[Tim Peters]

[Tim]
>> Because new-style classes create cycles that Py_Finalize() doesn't
>> clean up, it may make analysis easier to stick a PyGC_Collect() call
>> (or two!  repeat until it returns 0) inside the loop now.

[Martin]
> I'm shy to do this: the comment in Py_Finalize suggests that things
> will break if there is a "late" garbage collection.

Putting a collection call inside an initialize/finalize loop isn't
doing it late, it's doing it early.  If we can't collect cyclic trash
after Py_Initialize(), that would be a showstopper for apps embedding
Python "in a loop"!  There's either nothing to fear here, or Python
has a very bad bug.

Are you thinking of this comment?:

	/* Collect final garbage.  This disposes of cycles created by
	 * new-style class definitions, for example.
	 * XXX This is disabled because it caused too many problems.  If
	 * XXX a __del__ or weakref callback triggers here, Python code has
	 * XXX a hard time running, because even the sys module has been
	 * XXX cleared out (sys.stdout is gone, sys.excepthook is gone, etc).
	 * XXX One symptom is a sequence of information-free messages
	 * XXX coming from threads (if a __del__ or callback is invoked,
	 * XXX other threads can execute too, and any exception they encounter
	 * XXX triggers a comedy of errors as subsystem after subsystem
	 * XXX fails to find what it *expects* to find in sys to help report
	 * XXX the exception and consequent unexpected failures).  I've also
	 * XXX seen segfaults then, after adding print statements to the
	 * XXX Python code getting called.
	 */

I wrote that, and think it's pretty clear:  after PyImport_Cleanup(),
so little of the interpreter still exists that _any_ problem while
running Python code has a way of turning into a fatal problem.  For
example, internal error mechanisms that fetch things from the sys
module (like excepthook or stdout) are usually (always?) careful to
check whether the fetched things are NULL, but wander into lala-land
when Py_None comes back (as it does after PyImport_Cleanup()
"Nones-out" everything in sys).

But call Py_Initialize() again, and everything (including the sys
module) should become usable again.

...

>>> This totals to 360, which is for some reason higher than the numbers
>>> I get when counting the objects on the global list of objects.

>> How much higher?

> Well, I counted an increase of 156 objects on the "all objects"
> list, and an increase of 360 according to the COUNT_ALLOCS numbers.
> The first number was without COUNT_ALLOCS being defined, though.
>
> Anyway, thanks for your comments. I'll try to look at this from
> time to time, maybe I can resolve some of the leaks.

Could you check in the code you're using?  It would be useful to share
this, and improve it over time.  Of course it's easy to write a C
program that calls Py_Initialize() and Py_Finalize() in a loop, but it
gets real tedious real fast to add analysis code sufficient to truly
help track down leaks.  I'd be happiest to contribute future hints in
the form of working C code :-)

Heck, if we got the leaks down to 0 on second and subsequent calls, we
could even make a standard test out of it to ensure it stays that way.