[IronPython] Memory leaks in Ipy 2.6

[Dino Viehland]

This is definitely a bug - we're leaking our stack frames on the CLR's finalizer thread when we swallow exceptions while closing the generator.  I think this is the correct fix but I haven't fully tested it yet, in our generator we have the code below but it's missing the assignment of null to DynamicStackFrames:

            try {
                @throw(new GeneratorExitException());

                // Generator should not have exited normally.
                throw new RuntimeException("generator ignored GeneratorExit");
            } catch (StopIterationException) {
                // Ignore
                ExceptionHelpers.DynamicStackFrames = null;  // adding this fixes it
            } catch (GeneratorExitException) {
                // Ignore
                ExceptionHelpers.DynamicStackFrames = null; // adding this fixes it
            }

This definitely fixes 1 and 2.  After running this 3 doesn't actually seem to leak memory when running repeatedly for me but after-before > 10 is True.  I'll take a closer look and make sure this is the correct fix after the new year when I'm back in the office.

From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg
Sent: Monday, December 28, 2009 10:33 AM
To: users at lists.ironpython.com
Subject: [IronPython] Memory leaks in Ipy 2.6

Hi,
Working with the new version I have encountered some problems which look like memory leaks to me.
I've written 3 test methods that reproduce the problems, and would appreciate your response.
Thanks
Problem 1
Occurs when you do the all of the  following

1.       Define a generator method

2.       Insert a try/except clause in the method

3.       define an inner method that uses some local variable

4.       Call the generator method without reaching the "StopIteration"
Looks like the local variable used by the inner method is never cleared.
This code reproduces the problem:
    def test_generator_memory_leak(self):
                """
                   Ipy 2.6 This test reproduces a memory leak when calling a generator method without reaching the end.
                """
                def coroutine():
                    try: pass
                    except: pass
                    just_numbers = range(1,1000)
                    def inner_method():
                                return just_numbers
                    yield None
                    yield None

                from System import GC
                def get_memory():
                    for _ in xrange(4):
                                GC.Collect()
                                GC.WaitForPendingFinalizers()
                    return GC.GetTotalMemory(True)/1e6
                before = get_memory()
                for j in xrange(10000):
                    crt = coroutine()
                    crt.next()
                after = get_memory()
                self.assert_(after-before > 10,'There should be a memory leak in this case.before=%s after=%s' % (before,after))

Problem 2
The same as problem, just instead of defining an inner method, just call "eval" with any string
     def test_generator_memory_leak2(self):
                """
                   Ipy 2.6 This test reproduces a memory leak when calling a generator method without reaching the end.
                """
                def coroutine(b_bool):
                    try: pass
                    except: pass
                    if False: eval("")

                    just_numbers = range(1,1000)
                    yield None
                    yield None

                from System import GC
                def get_memory():
                    for _ in xrange(4):
                                GC.Collect()
                                GC.WaitForPendingFinalizers()
                    return GC.GetTotalMemory(True)/1e6
                before = get_memory()
                for j in xrange(10000):
                    crt = coroutine(False)
                    crt.next()

                after = get_memory()
                self.assert_(after-before > 10,'There should be a memory leak in this case.before=%s after=%s' % (before,after))

Problem 3
This is actually a our solution to problems 1,2. We noticed that if we end the iteration by using "throw" then the memory doesn't rise on subsequent instances. Still there some memory increase that depends on the size of the local variable which doesn't seem to go away:

     def test_generator_memory_leak2(self):
                """
                   Ipy 2.6 when exiting a generator method with an exception, some objects are never collected
                   This seems to be static to the type (the leak does not grow if we repeat the experiment
                """
                def coroutine():
                    just_numbers = range(1,1000000)
                    def inner_method():
                                return just_numbers
                    yield None
                    raise Exception("some exception") # comment out this line to make the test not work
                from System import GC
                def get_memory():
                    for _ in xrange(4):
                                GC.Collect()
                                GC.WaitForPendingFinalizers()
                    return GC.GetTotalMemory(True)/1e6
                before = get_memory()
                crt = coroutine()
                try:
                    crt.next()
                    crt.next()
                except:
                    pass
                crt = None
                after = get_memory()
                self.assert_(after-before > 10,'There should be a memory leak in this case.before=%s after=%s' % (before,after))


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20091228/962d7cc8/attachment.html>