when does python free memory?

Brian Kelley bkelley at wi.mit.edu
Wed Feb 6 11:11:49 EST 2002


Andrae Muys wrote:

>reanes at banta-im.com (Robert Eanes) wrote in message news:<10b3f29a.0202050840.7641923a at posting.google.com>...
>
>>Hi-
>>Reading the docs and searching the list archives, it seemed that
>>python should release memory as soon as the last reference to an
>>object is deleted.  However, I can't seem to prove this in practice. 
>>Look at this example:
>>
>
>This is true for 1.5.2, however later versions of python may use a
>garbage collected store for some/all variables.  I don't know enough
>about python's gc to discuss this.  (ps. anyone got a good reference
>on a discussion of the current CPython (de-)allocator?)
>
><SNIP sample code demonstrating retained heap>
>
>>So apparently python knows that the memory formerly allocated to z is
>>available, but it doesn't tell the OS.  Is this the case, and if so,
>>is there any way around it?  I have an application that needs to run
>>80-150 python processes that can stay around for 15min to an hour, so
>>I'd like them not to hold onto the maximum amount of memory that they
>>use throughout the whole life of the process.
>>
Python running under windows does release memory back to the O/S so it 
is indeed a malloc/free issue.  I got over some of these issues running 
on Linux by running large memory consuming tasks in their own process 
and not their own thread.  Each process essentially had a parent process 
which controlled the I/O back to the scheduling program and a child 
process for doing the heavy lifting.

The child processes were started using calls popen2.  The drawbacks are 
that data has to be transferred back and forth through the read/write 
pipes or through some other mechanism and that the interaction with the 
child was very coarse grained.  Benefits included that the child process 
could be killed at any time and when it finished it gave memory back to 
the O/S, they included a simple callback mechanisms with a 
model-observer pattern for simple interactions and they didn't have the 
dreaded python-thread performance hit which sometimes was as high as 35% 
for gui based applications.

We eventually wrapped these up into a bunch of classes that we were 
derived from what we called LongRunningTasks that behaved similarly to 
threads but were able to be killed.

I gave a portion of a talk at the O'Reilly conference last year that 
covered a part of the issues involved.  Since then, I haven't seemed to 
generate enough interest for pursuing these psuedo-threads.  I spent a 
little bit of time porting them to windows as well.

Mitch Chapman wrote most of the code I used back at Bioreason, maybe I 
can convince him to help me formalize a package for doing this type of 
thing.

Brian Kelley
Whitehead Institute for Biomedical Research





More information about the Python-list mailing list