[Python-bugs-list] [ python-Bugs-543148 ] Memory leak with stackframes + inspect
noreply@sourceforge.net
noreply@sourceforge.net
Tue, 23 Apr 2002 14:21:41 -0700
Bugs item #543148, was opened at 2002-04-12 14:48
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=543148&group_id=5470
Category: Documentation
Group: None
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Bernhard Herzog (bernhard)
Assigned to: Fred L. Drake, Jr. (fdrake)
Summary: Memory leak with stackframes + inspect
Initial Comment:
The following program leaks memory on Python 2.1.3:
import inspect
def leak():
frame = inspect.currentframe()
while 1:
leak()
(On Linux at least the process size grows *very*
quickly!)
System:
Python 2.1.3 (#1, Apr 12 2002, 20:09:56)
[GCC 2.95.4 20011006 (Debian prerelease)] on linux2
It has no memory problems with Python 2.2.
Workaround: del frame in leak.
The docs at least should mention this.
----------------------------------------------------------------------
>Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2002-04-23 17:21
Message:
Logged In: YES
user_id=3066
Added text & example based on previous comment to
Doc/lib/libinspect.tex revisions 1.3.2.1, 1.10.6.1, and 1.11.
----------------------------------------------------------------------
Comment By: Bernhard Herzog (bernhard)
Date: 2002-04-13 10:26
Message:
Logged In: YES
user_id=2369
For the documentation, perhaps a note in section 3.11.4,
"The interpreter stack", might be appropriate. Something
like this:
In Python 2.1 and earlier, stackframes stored directly or
indirectly in local variables can easily cause referency
cycles the garbage collector can't collect, leading to
memory leaks. To avoid this, it's a good idea to
explicitly remove the cycle in a finally: clause. For
example:
def handle_stackframe_without_leak():
frame = inspect.currentframe()
try:
# do something with the frame
finally:
del frame
The situation here is very similar to sys.exc_info and the
traceback in its return value.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-04-13 01:25
Message:
Logged In: YES
user_id=31435
The slower leak in current CVS got plugged via bounding the
frameobject free_list, in
Objects/frameobject.c; new revision: 2.61
Also backported to the 2.2 branch.
Reassigned to Fred for 2.1, in case he can figure out what
kind of doc change might help.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-04-12 20:03
Message:
Logged In: YES
user_id=31435
Neil, are there other gcable containers that use free
lists? Offhand I could only think of tuples, but they
already put a bound on the their free list size
(MAXSAVEDTUPLES). Of course I agree it is a general
problem.
Another approach could expose part of the guts of
_PyObject_GC_Malloc, and tell free-list addicts (FLAs) they
have to call that on every "virtual" allocation; similarly
for virtual deallocation. The "excess" approach remains, I
think, a clever and solid one, and from that view "the
problem" is that the FLAs simply don't participate in it.
OTOH, FLAs are using a free-list because they're keen to
make allocation as fast as possible, so making newing
external calls is unattractive on that ground.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-04-12 19:05
Message:
Logged In: YES
user_id=31435
Reassigned to me.
Bernhard, can you suggest where the docs could say
something in 2.1 that would do you any good? And/or what
they could say that would help? Documenting cases where
cycles can cause leaks hasn't appeared to do much good over
the years -- if someone doesn't already know that assigning
the current frame to a local vrbl in that frame causes a
cycle, or why such a cycle leaks before 2.2, they're not
easy things to get across.
----------------------------------------------------------------------
Comment By: Neil Schemenauer (nascheme)
Date: 2002-04-12 18:54
Message:
Logged In: YES
user_id=35752
Sounds like a sensible solution. I think this is a
problem with all GC collected objects that use a free list.
I can't think of a nice general solution for it though (other
than removing the other free lists :).
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-04-12 18:39
Message:
Logged In: YES
user_id=31435
OK, it *is* the frameobject free_list that's screwing us
here. Disable that mechanism, and the fleak2.py output is
steady over time, and the process doesn't grow over time.
gc is triggered by an excess of allocations over
deallocations, by some delta N. After the first N frames
in cycles are allocated, gc cleans them up, but they all go
into the free_list. Then it takes N+N frames in cycles
before gc is triggered again (the first N come out of the
free_list so don't bump the allocation count at all), and
all N+N are returned to the free_list. Then it takes N+N+N
frames in cycles before gc is triggered again, etc. Over
time, the number of frames in the free_list grows without
bound.
I'm thinking about putting a bound on it. That is, if
frame_dealloc sees there are already K frames on free_list,
free the memory instead of keeping it devoted to frames
forever.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-04-12 18:27
Message:
Logged In: YES
user_id=31435
Ack, I'm hallucinating -- forget fleak.py. What it
actually shows is that we're *not* leaking frames(!).
Something is still very strange, but I don't know what.
Staring at the output from (the much simpler) fleak2.py
shows that the 1st generation keeps growing over time,
which is weird by itself. gc believes it reclaims
everything unreachable each time it triggers too, yet the
process grows about 50MB/minute on my box.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-04-12 18:00
Message:
Logged In: YES
user_id=31435
The attached fleak.py seems to show that, under 2.2 and
current CVS, we're leaking one frame per top-level call.
It prints
gc: collecting generation 2...
gc: objects in each generation: 0 0 21971
gc: done, 20000 unreachable, 0 uncollectable.
20000
[<frame object at 0x00755B90>]
{<type 'frame'>: 20000}
However, stranger, if you set VARIANT=1 at the top, the
output changes to
gc: collecting generation 2...
gc: objects in each generation: 0 0 2070
gc: done, 99 unreachable, 0 uncollectable.
99
[<frame object at 0x0077C770>]
{<type 'frame'>: 99}
So (this will make sense if you look at the code), the
*frequency* of gc determines how much stuff is considered
unreachable in the end. Disabling the frameobject
free_list didn't appear to make any difference to this.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-04-12 15:16
Message:
Logged In: YES
user_id=31435
Yes, when you create a cyclic structure involving frames in
2.1, it plain leaks -- frames aren't looked at by the
cyclic garbage detector in 2.1. Frames were added to
cyclic gc collection in 2.2.
However, it sure looks to me like this program *still*
leaks in 2.2 (and current CVS), just a lot slower (I've
watched it grow to over 30MB on Windows, with no sign of
ever stopping). But if I call gc.collect() periodically in
the "while 1:" (every 100th time) it doesn't leak at all
(and stays at about 2MB). Assigning to Neal in the hopes
that it will be instantly obvious to him why gc isn't
happening without being forced -- or perhaps why gc is
happening but is ineffective in this case. Maybe the
frameobject free_list is interfering?
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=543148&group_id=5470