[Python-bugs-list] [ python-Bugs-543148 ] Memory leak with stackframes + inspect

noreply@sourceforge.net noreply@sourceforge.net
Fri, 12 Apr 2002 15:39:32 -0700


Bugs item #543148, was opened at 2002-04-12 14:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=543148&group_id=5470

Category: None
Group: Python 2.1.2
Status: Open
Resolution: None
Priority: 5
Submitted By: Bernhard Herzog (bernhard)
Assigned to: Neil Schemenauer (nascheme)
Summary: Memory leak with stackframes + inspect

Initial Comment:
The following program leaks memory on Python 2.1.3:

import inspect

def leak():
    frame = inspect.currentframe()

while 1:
    leak()

(On Linux at least the process size grows *very*
quickly!)

System:
Python 2.1.3 (#1, Apr 12 2002, 20:09:56) 
[GCC 2.95.4 20011006 (Debian prerelease)] on linux2


It has no memory problems with Python 2.2.

Workaround: del frame in leak.
The docs at least should mention this.


----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2002-04-12 18:39

Message:
Logged In: YES 
user_id=31435

OK, it *is* the frameobject free_list that's screwing us 
here.  Disable that mechanism, and the fleak2.py output is 
steady over time, and the process doesn't grow over time.

gc is triggered by an excess of allocations over 
deallocations, by some delta N.  After the first N frames 
in cycles are allocated, gc cleans them up, but they all go 
into the free_list.  Then it takes N+N frames in cycles 
before gc is triggered again (the first N come out of the 
free_list so don't bump the allocation count at all), and 
all N+N are returned to the free_list.  Then it takes N+N+N 
frames in cycles before gc is triggered again, etc.  Over 
time, the number of frames in the free_list grows without 
bound.

I'm thinking about putting a bound on it.  That is, if 
frame_dealloc sees there are already K frames on free_list, 
free the memory instead of keeping it devoted to frames 
forever.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-12 18:27

Message:
Logged In: YES 
user_id=31435

Ack, I'm hallucinating -- forget fleak.py.  What it 
actually shows is that we're *not* leaking frames(!).

Something is still very strange, but I don't know what.  
Staring at the output from (the much simpler) fleak2.py 
shows that the 1st generation keeps growing over time, 
which is weird by itself.  gc believes it reclaims 
everything unreachable each time it triggers too, yet the 
process grows about 50MB/minute on my box.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-12 18:00

Message:
Logged In: YES 
user_id=31435

The attached fleak.py seems to show that, under 2.2 and 
current CVS, we're leaking one frame per top-level call.  
It prints

gc: collecting generation 2...
gc: objects in each generation: 0 0 21971
gc: done, 20000 unreachable, 0 uncollectable.
20000
[<frame object at 0x00755B90>]
{<type 'frame'>: 20000}

However, stranger, if you set VARIANT=1 at the top, the 
output changes to

gc: collecting generation 2...
gc: objects in each generation: 0 0 2070
gc: done, 99 unreachable, 0 uncollectable.
99
[<frame object at 0x0077C770>]
{<type 'frame'>: 99}

So (this will make sense if you look at the code), the 
*frequency* of gc determines how much stuff is considered 
unreachable in the end.  Disabling the frameobject 
free_list didn't appear to make any difference to this.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-12 15:16

Message:
Logged In: YES 
user_id=31435

Yes, when you create a cyclic structure involving frames in 
2.1, it plain leaks -- frames aren't looked at by the 
cyclic garbage detector in 2.1.  Frames were added to 
cyclic gc collection in 2.2.

However, it sure looks to me like this program *still* 
leaks in 2.2 (and current CVS), just a lot slower (I've 
watched it grow to over 30MB on Windows, with no sign of 
ever stopping).  But if I call gc.collect() periodically in 
the "while 1:" (every 100th time) it doesn't leak at all 
(and stays at about 2MB).  Assigning to Neal in the hopes 
that it will be instantly obvious to him why gc isn't 
happening without being forced -- or perhaps why gc is 
happening but is ineffective in this case.  Maybe the 
frameobject free_list is interfering?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=543148&group_id=5470