[IronPython] Memory leaks in Ipy 2.6

[Idan Zaltzberg]

Thanks for the fast reply.

Could you estimate when and how (in what version) the fix will be deployed?

Regardless, our current fix is to "throw" exception when we want the
generator to be disposed and catch it ourselves, is that the right way to
go?



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland
*Sent:* Monday, December 28, 2009 9:47 PM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] Memory leaks in Ipy 2.6



This is definitely a bug – we’re leaking our stack frames on the CLR’s
finalizer thread when we swallow exceptions while closing the generator.  I
think this is the correct fix but I haven’t fully tested it yet, in our
generator we have the code below but it’s missing the assignment of null to
DynamicStackFrames:



            try {

                @throw(new GeneratorExitException());



                // Generator should not have exited normally.

                throw new RuntimeException("generator ignored GeneratorExit"
);

            } catch (StopIterationException) {

                // Ignore

                ExceptionHelpers.DynamicStackFrames = null;  // adding this
fixes it

            } catch (GeneratorExitException) {

                // Ignore

                ExceptionHelpers.DynamicStackFrames = null; // adding this
fixes it

            }



This definitely fixes 1 and 2.  After running this 3 doesn’t actually seem
to leak memory when running repeatedly for me but after-before > 10 is
True.  I’ll take a closer look and make sure this is the correct fix after
the new year when I’m back in the office.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg
*Sent:* Monday, December 28, 2009 10:33 AM
*To:* users at lists.ironpython.com
*Subject:* [IronPython] Memory leaks in Ipy 2.6



Hi,

Working with the new version I have encountered some problems which look
like memory leaks to me.

I've written 3 test methods that reproduce the problems, and would
appreciate your response.

Thanks

*Problem 1*

Occurs when you do the all of the  following

1.       Define a generator method

2.       Insert a try/except clause in the method

3.       define an inner method that uses some local variable

4.       Call the generator method without reaching the "StopIteration"

Looks like the local variable used by the inner method is never cleared.

This code reproduces the problem:

    def test_generator_memory_leak(self):

                """

                   Ipy 2.6 This test reproduces a memory leak when calling a
generator method without reaching the end.

                """

                def coroutine():

                    try: pass

                    except: pass

                    just_numbers = range(1,1000)

                    def inner_method():

                                return just_numbers

                    yield None

                    yield None



                from System import GC

                def get_memory():

                    for _ in xrange(4):

                                GC.Collect()

                                GC.WaitForPendingFinalizers()

                    return GC.GetTotalMemory(True)/1e6

                before = get_memory()

                for j in xrange(10000):

                    crt = coroutine()

                    crt.next()

                after = get_memory()

                self.assert_(after-before > 10,'There should be a memory
leak in this case.before=%s after=%s' % (before,after))



*Problem 2*

The same as problem, just instead of defining an inner method, just call
"eval" with any string

     def test_generator_memory_leak2(self):

                """

                   Ipy 2.6 This test reproduces a memory leak when calling a
generator method without reaching the end.

                """

                def coroutine(b_bool):

                    try: pass

                    except: pass

                    if False: eval("")



                    just_numbers = range(1,1000)

                    yield None

                    yield None



                from System import GC

                def get_memory():

                    for _ in xrange(4):

                                GC.Collect()

                                GC.WaitForPendingFinalizers()

                    return GC.GetTotalMemory(True)/1e6

                before = get_memory()

                for j in xrange(10000):

                    crt = coroutine(False)

                    crt.next()



                after = get_memory()

                self.assert_(after-before > 10,'There should be a memory
leak in this case.before=%s after=%s' % (before,after))



*Problem 3*

This is actually a our solution to problems 1,2. We noticed that if we end
the iteration by using "throw" then the memory doesn’t rise on subsequent
instances. Still there some memory increase that depends on the size of the
local variable which doesn’t seem to go away:



     def test_generator_memory_leak2(self):

                """

                   Ipy 2.6 when exiting a generator method with an
exception, some objects are never collected

                   This seems to be static to the type (the leak does not
grow if we repeat the experiment

                """

                def coroutine():

                    just_numbers = range(1,1000000)

                    def inner_method():

                                return just_numbers

                    yield None

                    raise Exception("some exception") # comment out this
line to make the test not work

                from System import GC

                def get_memory():

                    for _ in xrange(4):

                                GC.Collect()

                                GC.WaitForPendingFinalizers()

                    return GC.GetTotalMemory(True)/1e6

                before = get_memory()

                crt = coroutine()

                try:

                    crt.next()

                    crt.next()

                except:

                    pass

                crt = None

                after = get_memory()

                self.assert_(after-before > 10,'There should be a memory
leak in this case.before=%s after=%s' % (before,after))
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20091229/052a6996/attachment.html>