when does python free memory?
Brian Kelley
bkelley at wi.mit.edu
Wed Feb 6 11:11:49 EST 2002
Andrae Muys wrote:
>reanes at banta-im.com (Robert Eanes) wrote in message news:<10b3f29a.0202050840.7641923a at posting.google.com>...
>
>>Hi-
>>Reading the docs and searching the list archives, it seemed that
>>python should release memory as soon as the last reference to an
>>object is deleted. However, I can't seem to prove this in practice.
>>Look at this example:
>>
>
>This is true for 1.5.2, however later versions of python may use a
>garbage collected store for some/all variables. I don't know enough
>about python's gc to discuss this. (ps. anyone got a good reference
>on a discussion of the current CPython (de-)allocator?)
>
><SNIP sample code demonstrating retained heap>
>
>>So apparently python knows that the memory formerly allocated to z is
>>available, but it doesn't tell the OS. Is this the case, and if so,
>>is there any way around it? I have an application that needs to run
>>80-150 python processes that can stay around for 15min to an hour, so
>>I'd like them not to hold onto the maximum amount of memory that they
>>use throughout the whole life of the process.
>>
Python running under windows does release memory back to the O/S so it
is indeed a malloc/free issue. I got over some of these issues running
on Linux by running large memory consuming tasks in their own process
and not their own thread. Each process essentially had a parent process
which controlled the I/O back to the scheduling program and a child
process for doing the heavy lifting.
The child processes were started using calls popen2. The drawbacks are
that data has to be transferred back and forth through the read/write
pipes or through some other mechanism and that the interaction with the
child was very coarse grained. Benefits included that the child process
could be killed at any time and when it finished it gave memory back to
the O/S, they included a simple callback mechanisms with a
model-observer pattern for simple interactions and they didn't have the
dreaded python-thread performance hit which sometimes was as high as 35%
for gui based applications.
We eventually wrapped these up into a bunch of classes that we were
derived from what we called LongRunningTasks that behaved similarly to
threads but were able to be killed.
I gave a portion of a talk at the O'Reilly conference last year that
covered a part of the issues involved. Since then, I haven't seemed to
generate enough interest for pursuing these psuedo-threads. I spent a
little bit of time porting them to windows as well.
Mitch Chapman wrote most of the code I used back at Bioreason, maybe I
can convince him to help me formalize a package for doing this type of
thing.
Brian Kelley
Whitehead Institute for Biomedical Research
More information about the Python-list
mailing list