[Python-bugs-list] [Bug #128040] memory leak with multiple Py_Initialize/Py_Finalize calls

[noreply@sourceforge.net]

Bug #128040, was updated on 2001-Jan-08 07:00
Here is a current snapshot of the bug.

Project: Python
Category: Python Interpreter Core
Status: Closed
Resolution: Fixed
Bug Group: None
Priority: 5
Submitted by: pbmtl
Assigned to : bwarsaw
Summary: memory leak with multiple Py_Initialize/Py_Finalize calls

Details: I run script that use the win32 extension embedded in a server. 
When I call the Py_Finalize method, the memory is not free completely. 
Each time I run a script, my server uses more and more memory.
I think the Patch #101713 talk about this problem.  It seems that python
have problem handling multiple Py_Initialize and Py_Finalise when extension
are used.

Follow-Ups:

Date: 2001-Jan-23 09:54
By: bwarsaw

Comment:
I took Guido's Demo/embed/loop.c test, compiled it with Insure, and ran a
series of tests using a simple `pass' loop.  There were definitely objects
that got caught in cycles and never freed, but also didn't technically leak
(in the sense that there were still C pointers to the blocks).  

There was a big problem with the exceptions.c module, and I've now fixed
that.  Now the loop.c test does not grow with each interation of the loop
(modulo a few tiny leaks in Jeremy's still experimental symbol table stuff
in the compiler).  The interpreter still keeps some objects alive past
Py_Finalize(), e.g. the interned strings dictionary, but those are never
reallocated at the next Py_Initialize() so they should be fine.

Closing this bug report.
-------------------------------------------------------

Date: 2001-Jan-23 02:18
By: tim_one

Comment:
tregnago, that's what Insure++ is.
-------------------------------------------------------

Date: 2001-Jan-23 02:02
By: tregnago

Comment:
In Borland C++, under MSWin, we have a tool called CodeGuard that it's very
useful to check out this kind of problems. Can't you simply do a Python
debug session with a similar tool?
-------------------------------------------------------

Date: 2001-Jan-19 15:11
By: gvanrossum

Comment:
One for Barry and Insure++!
-------------------------------------------------------

Date: 2001-Jan-10 08:59
By: gvanrossum

Comment:
I agree with Mark that in the past we have been lax in addressing these
kinds of problems.  I do wish to fix these in the future!  It's just hard
to find all the leaks.  In the past we had reports where a simple loop
calling Py_Initialize() and Py_Finalize() with nothing in between would
leak; we've fixed these.  But now if you put real work in between it still
seems to leak.  I'll see if I can check in a test program in the Demo tree
which Barry can then work on using Insure++.

pbmtl, when you write that "the call to PyEval_CallObject will return an
error", can you submit a separate bug report for that with enough details
(e.g. C source code) so that we can reproduce the problem?  This seems to
be a separate bug, unrelated to leakage!

-------------------------------------------------------

Date: 2001-Jan-10 06:17
By: pbmtl

Comment:
I agree with you that it's not that important when script are executed
using PythonWin for example, because when PythonWin is closed, the memory
is recover.  

But if you use python embedded in a server on a remote computer that is
never supposed to be stopped, it's a real problem.  The memory usage
increase rapidly, Hundreds KB at each execution in my case. 

In fact, I can't call PyInitialize and PyFinalize for each script because
if I do, the second script to be executed will not work correctly (the call
to PyEval_CallObject will return an error).  I use the Py_NewInterpreter
and the Py_EndInterpreter.


For my part, I think the script virtual machine should be responsable of
the memory created by the scripts.  Otherwise, there is no garanty that the
PyFinalyze will free the memory correctly.  

Thanks

-------------------------------------------------------

Date: 2001-Jan-09 21:20
By: tim_one

Comment:
Thank you, Mark!  Since nobody is going to volunteer to do all this stuff,
and everyone else will try to stop them if they did <wink>, I reassigned it
to Guido -- it's never going to get done if he doesn't push for it (and if
he doesn't think it's worth the effort, that's fine too, but that's his
call).

-------------------------------------------------------

Date: 2001-Jan-09 17:11
By: mhammond

Comment:
I dont believe this is Windows specific, nor specific to the Win32
extensions.  I can confirm that Python does leak with multiple Init/Terms,
as I had to change the COM extensions to only initialize once (and hence
never finalize).  It is definitely related to the referenced patch
(#101713)

This has been raised before, but there doesnt appear much enthusiasm for a
fix.  As far as I can tell, it would involve having Py_Finalize() correctly
release _all_ memory allocated by Py_Initialize().  However, I have seen
numerous people state that "one off" allocations that are not freed are not
worth finding and freeing - however, if this bug wants a fix, then we must
track them all down.

I further note that the problem is not only in the Python core, but in
extension modules.  Modules need a reliable way to be told that Python is
finalizing so they can free their own "one off" allocations.  Various hacks
have been proposed for this (eg, rely on ref-count semantics to ensure that
a special object is destroyed as the module dict is cleared) but nothing
"reasonable" has been proposed.

Really not sure what I can do here.  I am willing to do my bit in patching
up the leaks, but would want a (fuzzy) commitment from python-dev that I
wouldnt be doing it alone, and that it is seen as important.

I have updated the bug description and platform appropriately, but haven't
reallocated a new person.
-------------------------------------------------------

Date: 2001-Jan-09 14:38
By: tim_one

Comment:
Reassigned to MarkH.  Mark, sound familiar?

-------------------------------------------------------

Date: 2001-Jan-08 18:02
By: gvanrossum

Comment:
Assigned to Tim.  First task would be to request needed infromation from
submitter.

-------------------------------------------------------

For detailed info, follow this link:
http://sourceforge.net/bugs/?func=detailbug&bug_id=128040&group_id=5470