[ python-Bugs-1445210 ] embedding Python causes memory leaks

[SourceForge.net]

Bugs item #1445210, was opened at 2006-03-08 00:20
Message generated for change (Comment added) made by loewis
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1445210&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: None
>Status: Closed
>Resolution: Wont Fix
Priority: 5
Submitted By: Andrew Trevorrow (andykt)
Assigned to: Nobody/Anonymous (nobody)
Summary: embedding Python causes memory leaks

Initial Comment:
[This bug has been submitted by others but for some reason it
has been marked Closed.  I consider it to be an extremely serious
bug -- if I can't solve it I'm going to have to abandon Python as
my app's scripting language, even though I've fallen in love!]

I've added Python script support to my cross-platfom wxWidgets app
so that users can run .py scripts from within the app to automate the
GUI and do other fancy things.  It all works very nicely, except for
one nasty problem: *every* time a script is run there is a memory leak,
usually small (about 10K) but sometimes massive (about 4MB in the 
case of one rather complicated script).

The problem occurs on both Mac OS 10.3.9 and Windows 2000.
I'm using Python 2.3 on the Mac and 2.4.2 on Windows.

Every time the user runs a script, my app makes these calls:
(I've removed a lot of irrelevant stuff.)

   Py_Initialize();
   PyRun_SimpleString("execfile('foo.py')");
   Py_Finalize();

It's definitely not a wxWidgets problem.  In fact it's quite easy to
see the memory leak using a simple command-line program:

#include <stdio.h>
#include <Python.h>
main(int argc, char *argv[])
{
   int i;
   for (i=0; i<1000; i++) {
      Py_Initialize();
      Py_Finalize();
      printf(".");
      if ((i+1) % 50 == 0) printf("\n");
   }
}

Note that it doesn't even execute a script.  If I run this program on
my Mac and watch its memory usage with Activity Monitor, I see a leak
of about 10K each time through the loop.  Similar result on Windows.

Curiously, on both machines, the Py_Finalize() call takes longer and
longer to complete whatever it's doing.  The above program takes a
few *minutes* to complete on my 400MHz Mac.

Andrew


----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2006-04-13 09:29

Message:
Logged In: YES 
user_id=21627

That the documentation claims Py_Finalize releases all
memory is a bug; I just fixed this in r45344. The original
problem cannot be fixed (atleast not until Python 3000);
closing it as "won't fix".

----------------------------------------------------------------------

Comment By: Andrew Trevorrow (andykt)
Date: 2006-03-10 06:43

Message:
Logged In: YES 
user_id=1281947

See http://evanjones.ca/python-memory.html for some
useful info.  Apparently the Python memory allocator never
releases memory back to the OS!  So if a complicated script
happens to consume 100MB of interpreter memory then
that amount is no longer available to the app in which
Python is embedded.  Even worse, if a script has a
(Python) memory leak then there's nothing the app can
do about it.  It would be great if wrapping each script
inside Py_Initialize/Py_Finalize could avoid all that.

There should be some way to tell Python "release all
the memory you've ever allocated and start again
with a clean slate".



----------------------------------------------------------------------

Comment By: Andrew Trevorrow (andykt)
Date: 2006-03-08 10:50

Message:
Logged In: YES 
user_id=1281947

Bloody hell -- sorry for the bad line breaks.
One day I'll figure out how to use sf properly!


----------------------------------------------------------------------

Comment By: Andrew Trevorrow (andykt)
Date: 2006-03-08 10:46

Message:
Logged In: YES 
user_id=1281947

> Why do you call Py_Initialize/Py_Finalize more than once?

How else do I tell Python to free up memory after each PyRun_SimpleString
call?

I want users to be able to run scripts many times from within my app.
If I just keep calling PyRun_SimpleString then my app will leak more and
more memory until it becomes unusable.

>From http://docs.python.org/api/embedding.html:

  Sometimes, it is desirable to ``uninitialize'' Python. For instance, the
  application may want to start over (make another call to Py_Initialize())
  or the application is simply done with its use of Python and wants to free
  all memory allocated by Python. This can be accomplished by calling
  Py_Finalize().

That's exactly what I want to do.  I want the interpreter to run a script
and then release all the resources used by that script.  Unfortunately,
Py_Finalize does *not* restore memory usage to what it was before the
Py_Initialize call.  I wouldn't mind if there was a one-off allocation
cost (the 1st time Py_Initialize is called), but my app is leaking more
memory *every* time a script is run!


----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-03-08 09:32

Message:
Logged In: YES 
user_id=33168

Why do you call Py_Initialize/Py_Finalize more than once? 
Why not do something like this (I'm kinda mixing C and
Python for convenience):

 /* startup */
 Py_Initialize();

 /* do whatever */
 while (moreFiles()) {
   PyRun_SimpleString("execfile('%s')" % nextFile());
   /* do whatever */
 }

 /* shutdown */
 Py_Finalize();


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1445210&group_id=5470