[Python-Dev] Py_Finalize does not release all memory, not even closely

["Martin v. Löwis"]

Tim Peters wrote:
> Putting a collection call inside an initialize/finalize loop isn't
> doing it late, it's doing it early.  If we can't collect cyclic trash
> after Py_Initialize(), that would be a showstopper for apps embedding
> Python "in a loop"!  There's either nothing to fear here, or Python
> has a very bad bug.

Right. I did that, and it collects 308 objects after the first call
in the second "round" of Py_Initialize/Py_Finalize, and then no
additional objects.

However, I don't think that helps much: Py_Finalize will call
PyGC_Collect(), anyway, and before any counts are made.

> Are you thinking of this comment?:

Yes; I was assuming you suggested to enable that block of code.

> I wrote that, and think it's pretty clear:  after PyImport_Cleanup(),
> so little of the interpreter still exists that _any_ problem while
> running Python code has a way of turning into a fatal problem.

Right. I still haven't tried it, but it might be that, after a plain
Py_Initialize/Py_Finalize sequence, no such problems will occur,
and that it would be safe to call it in this specific case.

> Could you check in the code you're using?

I had to modify code in ways that shouldn't be checked in, e.g.
by putting API calls into _Py_PrintReferenceAddresses, even
though the comment says it does't call any API. When I get to
clean this up, I'll check it in.

With some debugging, I now found a "leak" that contributes
to quite some of these garbage objects: Each round of
Py_Initialize/Py_Finalize will leave a CodecInfo type
behind. I think it comes from this block of code

	/* Note that as of Python 2.2, heap-allocated type objects
	 * can go away, but this code requires that they stay alive
	 * until program exit.  That's why we're careful with
	 * refcounts here.  type_list gets a new reference to tp,
	 * while ownership of the reference type_list used to hold
	 * (if any) was transferred to tp->tp_next in the line above.
	 * tp is thus effectively immortal after this.
	 */
	Py_INCREF(tp);

so that this "leak" would only exist if COUNT_ALLOCS is defined.

I would guess that even more of the leaking type objects (16 per
round) can be attributed to this. This completely obstructs
measurements, and could well explain why the number of leaked
objects is so much higher when COUNT_ALLOCS is defined. OTOH,
I can see why "this code requires that they stay alive".

Any ideas on how to solve this dilemma? Perhaps the type_list
could be a list of weak references, so that the types do have
a chance to go away when the last instance disappears?

Regards,
Martin