[Patches] [ python-Patches-439364 ] GC for frames and generators
noreply@sourceforge.net
noreply@sourceforge.net
Mon, 23 Jul 2001 09:33:47 -0700
Patches item #439364, was opened at 2001-07-07 15:32
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=439364&group_id=5470
Category: core (C code)
Group: None
>Status: Closed
Resolution: Accepted
Priority: 5
Submitted By: Neil Schemenauer (nascheme)
Assigned to: Neil Schemenauer (nascheme)
Summary: GC for frames and generators
Initial Comment:
For me, test_generators leaks, with or without this
patch. The number of objects tracked by the GC does
not increase but memory usage does. There must be
other objects creating cycles that are not tracked by
the GC.
I haven't profiled the effect this change. I have
another patch in the works that should minimize the
performance hit.
Please examine frame_traverse and frame_clear closely.
I'm not 100% sure that I am chasing all the references
necessary to find cycles or that I am dropping enough
to break them.
----------------------------------------------------------------------
>Comment By: Neil Schemenauer (nascheme)
Date: 2001-07-23 09:33
Message:
Logged In: YES
user_id=35752
Closing patch since it was checked in a while ago.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-07-11 18:41
Message:
Logged In: YES
user_id=31435
Accepted, and back to Neil for checkin. I see no leaks
anymore. Thank you!
Disgustingly straightforward, wasn't it <wink>? The fact
that these cycles were so painful to track down (despite
being straightforward!) convinces me there's no future in
overlooking the leaks but trying to rationalize them away
on performance grounds. The patch makes necessary
progress, and we can worry about reclaiming performance as
a separate project (btw, that's easier for me if this is
checked in so there's a new baseline to work against).
----------------------------------------------------------------------
Comment By: Neil Schemenauer (nascheme)
Date: 2001-07-11 13:31
Message:
Logged In: YES
user_id=35752
I believe this patch does the trick. The performance hit
is larger than I would like however.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-07-10 20:45
Message:
Logged In: YES
user_id=31435
The newly attached bmoleak.py shows a similar leak without
any generators, involving seqiterobjects alone.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-07-09 21:34
Message:
Logged In: YES
user_id=31435
I expect what's going on here is the obvious <wink> thing:
clearing the LazyList dict stops the leak, but the dict
only holds two things: a vanilla list of integers, and a
bound method object. Since a list of ints can't cause a
cycle, it must be the latter; and, indeed, a quick glance
at methodobject.c shows GC doesn't chase PyCFunction_Type
(yet).
The bound method object in this case wraps the .next()
method of a generator-iterator object, and there's a
cycle: the LazyList fib maps fib.fetch to the fibgen()
generator-iterator's .next method, from which we can get to
the fibgen g-i, which in turn contains (indirect)
references back to fib thanks to passing iter(fib) to the
sum() and head() generators. Looks like breaking this by
magic requires adding both seqiterobjects and
PyCFunctionObjects to GC.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-07-09 14:08
Message:
Logged In: YES
user_id=31435
The attached fibleak.py is a self-contained massive leaker.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-07-09 13:48
Message:
Logged In: YES
user_id=31435
+ Performance hit is about a 2.5% slowdown. More than I
hoped, but less than I feared <wink>.
+ The leak after the patch is solely due to fun_tests.
Comment out the other lines in the __test__ dict, and on my
box it leaks about half a megabyte per second then -- very
dramatic, and impossible to ascribe to anything but a leak.
+ I don't know what causes the leak. Offhand my first
guess would be unchased pointers inside traceback objects.
Indeed, the first thing I'd *try* is adding tracebackobject
to GC too just to see whether that made the leak go away
(much easier than thinking about it <0.9 wink>).
----------------------------------------------------------------------
Comment By: Neil Schemenauer (nascheme)
Date: 2001-07-09 07:11
Message:
Logged In: YES
user_id=35752
Your right, it doesn't leak before the patch. The process
size goes up slowly be eventually stops increasing. After
the patch the size continues going up slowly. :-( I'll
try to find what is still leaking.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-07-08 21:40
Message:
Logged In: YES
user_id=31435
Thanks, Neil! I've been too overwhelmed with merging descr-
branch code to look at this until now.
Clarification, please: when you say "test_generators
leaks, with or without this patch", what exactly do you
mean by "leak"?
It does not leak for me before the patch (haven't yet tried
it with the patch), and this is what I mean-- exactly --
by "does not leak": I change a stock CVS build by
changing "if 0:" to "if 1:" in test_generators.py's
test_main, then let it run and run and run.
Now on Win98SE, the heap space *does* grow very slowly and
steadily, until it reaches about 3600Kb. Sometimes a
little more, sometimes a little less, depending on what
else I'm doing at the time. It takes about 5 minutes of
CPU time on a 866MHz box to reach that. But then it
*stays* at this point forever after (well, after another 10
minutes of CPU time it hasn't budged -- Win98SE probably
can't run forever <wink>).
This is presumably because Win98SE malloc is lazy about
coalescing free()'d memory until it nears a 4Mb boundary
(at which point it starts rearranging VM address space--
because 4Mb is all the address space the system allocates
to the initial heap --and it's very reluctant to do that).
So if you're seeing a very slow "leak" (are you?), you
*may* just be seeing a platform malloc reluctant to
defragment free()'d memory.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=439364&group_id=5470