[Python-Dev] Py_Finalize does not release all memory, not even closely

[Tim Peters]

[Martin v. Löwis]
> Several places in the documentation claim that Py_Finalize will
> release all memory:
>
> http://docs.python.org/api/embedding.html
>
> says that literally,

It's wrong ;-).

> and
>
> http://docs.python.org/api/initialization.html#l2h-778
>
> suggests it is a bug when memory is not freed in Py_Finalize.
>
> This has left people to believe that this is a bug:
>
> <https://sourceforge.net/tracker/index.php?func=detail&aid=1445210&group_id=5470&atid=105470>

Well, there may well be a bug (or multiple bugs) underlying that one
too.  It's one thing for Py_Finalize() not to release all memory (it
doesn't and probably never will), but it's not necessarily the same
thing if running Py_Initialize() ... Py_Finalize() repeatedly keeps
leaking more and more memory.

> However, I don't see any chance to make this promise even remotely.
> Objects allocated in extension modules, and held in global variables
> (e.g. socketmodule.c:socket_error, socket_herror, socket_gaierror,
> socket_timeout) will never be released, right?

Not unless the module has a finalization function called by
Py_Finalize() that frees such things (like PyString_Fini and
PyInt_Fini).

Other globals allocated via a

    static PyObject *someglobal = NULL;

...

    if (someglobal == NULL)
        someglobal = allocate_an_object_somehow();

pattern shouldn't contribute to continuing leaks across
Py_Initialize() ... Py_Finalize() loops.

> And because of the small objects allocator, their pool will remain
> allocated, right?  And, then, the arena.

Before Python 2.5, arenas are never freed, period.  In Python 2.5, an
arena will be freed if and only if it contains no allocated object by
the time Py_Finalize ends.

There may also be trash cycles that aren't collected during
Py_Finalize because I had to comment out Py_Finalize's second call to
PyGC_Collect(); new-style class objects are among the trash thingies
leaked (although if Py_Initialize() is called again, it's possible
that they'll get cleaned up by cyclic gc after all).

See Misc/SpecialBuilds.txt, section Py_TRACE_REFS, entry
PYTHONDUMPREFS, for a way to get a dump of all heap objects
Py_Finalize leaves alive.  I doubt that's been run in years; Guido and
I used it in 2.3b1 to cure some unreasonably high finalization leakage
at the time.

> So ISTM that invoking Py_Finalize after importing socket will yield
> atleast 256KiB garbage. Of course, that's not real garbage, because
> the next Py_Initialize'd interpreter will continue to allocate from
> the arenas.

I'm not clear on whether, e.g., init_socket() may get called more than
once if socket-slinging code appears in a Py_Initialize() ...
Py_Finalize().  If it doesn't, then, e.g., the unconditional

socket_gaierror = PyErr_NewException("socket.gaierror", ...);

won't contribute to ongoing leaks.  But if it does, then we'll
systematically leak the exception object on each loop trip.

> But still, the actual objects that the modules hold on to will not be reclaimed
> until the process terminates.

Without a _Fini() function called by Py_Finalize, that's correct.

> Please correct me if I'm wrong.

If you ever are, I will ;-)