[Python-bugs-list] [ python-Bugs-543148 ] Memory leak with stackframes + inspect

noreply@sourceforge.net noreply@sourceforge.net
Sat, 13 Apr 2002 07:26:34 -0700


Bugs item #543148, was opened at 2002-04-12 20:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=543148&group_id=5470

Category: Documentation
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Bernhard Herzog (bernhard)
Assigned to: Fred L. Drake, Jr. (fdrake)
Summary: Memory leak with stackframes + inspect

Initial Comment:
The following program leaks memory on Python 2.1.3:

import inspect

def leak():
    frame = inspect.currentframe()

while 1:
    leak()

(On Linux at least the process size grows *very*
quickly!)

System:
Python 2.1.3 (#1, Apr 12 2002, 20:09:56) 
[GCC 2.95.4 20011006 (Debian prerelease)] on linux2


It has no memory problems with Python 2.2.

Workaround: del frame in leak.
The docs at least should mention this.


----------------------------------------------------------------------

>Comment By: Bernhard Herzog (bernhard)
Date: 2002-04-13 16:26

Message:
Logged In: YES 
user_id=2369

For the documentation, perhaps a note in section 3.11.4,
"The interpreter stack", might be appropriate. Something
like this:

   In Python 2.1 and earlier, stackframes stored directly or
   indirectly in local variables can easily cause referency
   cycles the garbage collector can't collect, leading to
   memory leaks. To avoid this, it's a good idea to
   explicitly remove the cycle in a finally: clause. For
   example:

   def handle_stackframe_without_leak():
       frame = inspect.currentframe()
       try:
           # do something with the frame
       finally:
           del frame


The situation here is very similar to sys.exc_info and the
traceback in its return value.



----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-13 07:25

Message:
Logged In: YES 
user_id=31435

The slower leak in current CVS got plugged via bounding the 
frameobject free_list, in

Objects/frameobject.c; new revision: 2.61

Also backported to the 2.2 branch.

Reassigned to Fred for 2.1, in case he can figure out what 
kind of doc change might help.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-13 02:03

Message:
Logged In: YES 
user_id=31435

Neil, are there other gcable containers that use free 
lists?  Offhand I could only think of tuples, but they 
already put a bound on the their free list size 
(MAXSAVEDTUPLES).  Of course I agree it is a general 
problem.

Another approach could expose part of the guts of 
_PyObject_GC_Malloc, and tell free-list addicts (FLAs) they 
have to call that on every "virtual" allocation; similarly 
for virtual deallocation.  The "excess" approach remains, I 
think, a clever and solid one, and from that view "the 
problem" is that the FLAs simply don't participate in it.  
OTOH, FLAs are using a free-list because they're keen to 
make allocation as fast as possible, so making newing 
external calls is unattractive on that ground.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-13 01:05

Message:
Logged In: YES 
user_id=31435

Reassigned to me.

Bernhard, can you suggest where the docs could say 
something in 2.1 that would do you any good?  And/or what 
they could say that would help?  Documenting cases where 
cycles can cause leaks hasn't appeared to do much good over 
the years -- if someone doesn't already know that assigning 
the current frame to a local vrbl in that frame causes a 
cycle, or why such a cycle leaks before 2.2, they're not 
easy things to get across.

----------------------------------------------------------------------

Comment By: Neil Schemenauer (nascheme)
Date: 2002-04-13 00:54

Message:
Logged In: YES 
user_id=35752

Sounds like a sensible solution.  I think this is a 
problem with all GC collected objects that use a free list.
I can't think of a nice general solution for it though (other
than removing the other free lists :).

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-13 00:39

Message:
Logged In: YES 
user_id=31435

OK, it *is* the frameobject free_list that's screwing us 
here.  Disable that mechanism, and the fleak2.py output is 
steady over time, and the process doesn't grow over time.

gc is triggered by an excess of allocations over 
deallocations, by some delta N.  After the first N frames 
in cycles are allocated, gc cleans them up, but they all go 
into the free_list.  Then it takes N+N frames in cycles 
before gc is triggered again (the first N come out of the 
free_list so don't bump the allocation count at all), and 
all N+N are returned to the free_list.  Then it takes N+N+N 
frames in cycles before gc is triggered again, etc.  Over 
time, the number of frames in the free_list grows without 
bound.

I'm thinking about putting a bound on it.  That is, if 
frame_dealloc sees there are already K frames on free_list, 
free the memory instead of keeping it devoted to frames 
forever.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-13 00:27

Message:
Logged In: YES 
user_id=31435

Ack, I'm hallucinating -- forget fleak.py.  What it 
actually shows is that we're *not* leaking frames(!).

Something is still very strange, but I don't know what.  
Staring at the output from (the much simpler) fleak2.py 
shows that the 1st generation keeps growing over time, 
which is weird by itself.  gc believes it reclaims 
everything unreachable each time it triggers too, yet the 
process grows about 50MB/minute on my box.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-13 00:00

Message:
Logged In: YES 
user_id=31435

The attached fleak.py seems to show that, under 2.2 and 
current CVS, we're leaking one frame per top-level call.  
It prints

gc: collecting generation 2...
gc: objects in each generation: 0 0 21971
gc: done, 20000 unreachable, 0 uncollectable.
20000
[<frame object at 0x00755B90>]
{<type 'frame'>: 20000}

However, stranger, if you set VARIANT=1 at the top, the 
output changes to

gc: collecting generation 2...
gc: objects in each generation: 0 0 2070
gc: done, 99 unreachable, 0 uncollectable.
99
[<frame object at 0x0077C770>]
{<type 'frame'>: 99}

So (this will make sense if you look at the code), the 
*frequency* of gc determines how much stuff is considered 
unreachable in the end.  Disabling the frameobject 
free_list didn't appear to make any difference to this.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-12 21:16

Message:
Logged In: YES 
user_id=31435

Yes, when you create a cyclic structure involving frames in 
2.1, it plain leaks -- frames aren't looked at by the 
cyclic garbage detector in 2.1.  Frames were added to 
cyclic gc collection in 2.2.

However, it sure looks to me like this program *still* 
leaks in 2.2 (and current CVS), just a lot slower (I've 
watched it grow to over 30MB on Windows, with no sign of 
ever stopping).  But if I call gc.collect() periodically in 
the "while 1:" (every 100th time) it doesn't leak at all 
(and stays at about 2MB).  Assigning to Neal in the hopes 
that it will be instantly obvious to him why gc isn't 
happening without being forced -- or perhaps why gc is 
happening but is ineffective in this case.  Maybe the 
frameobject free_list is interfering?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=543148&group_id=5470