[IronPython] Please vote up this DLR memory leak bug

Dody Gunawinata empirebuilder at gmail.com
Tue Mar 3 12:55:09 CET 2009


OK, thanks for the tips. I'll dig around.

IronPython on ASP.Net is awesome - especially because it is instantly
hackable by non programmers (web designers). It has better
approachability than PHP or C# and low learning curve for simple web
programming. I notice similar thing with our XML on HTTP API. We have
people building a Visio client application and Excel (go figure) for
the CMS using VBScript just because they can concatenate   xml string
and send it using  built in http library and get their job done. No
design patterns, or OO or programming practice. Just code by casual
coders.

Dody G.

On Tue, Mar 3, 2009 at 12:13 AM, Dino Viehland <dinov at microsoft.com> wrote:
> Sorry for the slow response there's nothing that jumped out me immediately.  And the older versions make me inclined towards not investigating this too deeply.  Instead I talked to Jimmy about getting a new release of the ASP.Net support for IronPython out so hopefully we can do that soon.  That will be based off of IronPython 2.0.x which has the re-named namespaces for 3.5 compatability.
>
> If you want to drill into this more I'd suggest looking at all the strings that exist:
>
> !DumpHeap -mt 790fd8c4
>
> (note this number may change between runs) and:
>
> Dump all of the object arrays:
> !DumpHeap -mt 7912d8f8
>
> And then doing a:
>
> !da <array addr>
>
> On some of the individual arrays to see what they are holding onto.  Depending on what's in those arrays it may be an IronPython, DLR, or someone else's bugs.
>
> The presence of so many strings leads me to believe this could be a case of interned strings being a problem like in one of the repro cases, for example doing:
>
> for i in xrange(10000000):
>        exec 'def f' + str(i) + '(): pass'
>
> would be an example of this.
>
> You can also do !GCRoot on individual objects such as the strings.  If the strings are being kept alive by a Dictionary<string, int> or Dictionary<int, string> then it's our interning which is doing this.
>
> Let me know what you get from here and maybe there'll be something obvious or maybe I can have some other suggestions.
>
>
>> -----Original Message-----
>> From: users-bounces at lists.ironpython.com [mailto:users-
>> bounces at lists.ironpython.com] On Behalf Of Dody Gunawinata
>> Sent: Thursday, February 26, 2009 12:07 AM
>> To: Dino Viehland
>> Cc: Discussion of IronPython
>> Subject: Re: [IronPython] Please vote up this DLR memory leak bug
>>
>> Microsoft.Scripting2.Core is a namespace copy of the IP source code -
>> based on IP 2.0 Beta 3 - just before one version that introduced
>> incompatibility with .Net 3.5
>>
>> We are using ASP.Net with IronPython support based on July CTP 2007
>> (which I think ran on IP 2.0 Alpha X). The latest drop of IP for
>> ASP.Net was based on IP 2.0 Beta 4 which contained the .Net 3.5
>> incompatibility which makes in unusable for our application.
>>
>> So my application has 2 copies of IP - one that runs the asp.net
>> dynamic support and the other that runs the scripting support (with 2
>> suffix in the namespace). It's a stupid situation but we rely a IP for
>> ASP.Net and that piece of technology has not been brought up to date
>> to a working state for .Net 3.5 framework- and our web productions
>> people rely on the software so it's full speed ahead regardless of the
>> technology limitation.
>>
>> This memory leak might be gone if we have an up to date IP for ASP.Net
>> but one never know so here's the last 25 lines of the heap dump. I
>> hope this is useful.
>>
>>
>>
>> 79104368     3939        94536 System.Collections.ArrayList
>> 7911f030     1708        95648 System.Reflection.Emit.DynamicMethod
>> 7911f400     2999        95968 System.Reflection.Emit.SignatureHelper
>> 109cbc9c     2516       100640
>> System.Linq.Expressions2.ConditionalExpression
>> 1085c930     3181       101792 System.Scripting.SourceSpan
>> 109c4594     3244       103808
>> System.Linq.Expressions2.ParameterExpression
>> 10f63bcc     1237       110188
>> System.Collections.Generic.Dictionary`2+Entry[[System.Linq.Expressions2
>> .Expression,
>> Microsoft.Scripting2.Core],[System.Linq.Expressions2.CompilerScope+Stor
>> age,
>> Microsoft.Scripting2.Core]][]
>> 79135014        3       114968
>> System.Collections.Generic.Dictionary`2+Entry[[System.Int32,
>> mscorlib],[System.String, mscorlib]][]
>> 10f60a2c     2260       117520
>> System.Collections.Generic.Dictionary`2[[System.Linq.Expressions2.Expre
>> ssion,
>> Microsoft.Scripting2.Core],[System.Object, mscorlib]]
>> 109cfb1c     2736       120384
>> System.Linq.Expressions2.BinaryExpression
>> 10f6145c     1743       125700
>> System.Collections.Generic.Dictionary`2+Entry[[System.Linq.Expressions2
>> .Expression,
>> Microsoft.Scripting2.Core],[System.Object, mscorlib]][]
>> 109c4e4c     3164       126560 System.Linq.Expressions2.ScopeExpression
>> 109c4274     4142       132544
>> System.Linq.Expressions2.VariableExpression
>> 109caf7c     3735       134460 System.Linq.Expressions2.UnaryExpression
>> 10f5cd24     3762       135432
>> System.Linq.Expressions2.StackSpiller+ChildRewriter
>> 106fa09c       40       138784
>> System.Collections.Generic.Dictionary`2+Entry[[Microsoft.Scripting.Symb
>> olId,
>> Microsoft.Scripting],[Microsoft.Scripting.DynamicMixin+SlotInfo,
>> Microsoft.Scripting]][]
>> 79111038     2585       144760 System.Reflection.RuntimePropertyInfo
>> 79110680     3086       148128 System.Signature
>> 106f63bc     4183       150588 Microsoft.Scripting.BuiltinFunction
>> 109cbb3c     4327       155772
>> System.Linq.Expressions2.MemberExpression
>> 791336d8       37       162960
>> System.Collections.Generic.Dictionary`2+Entry[[System.String,
>> mscorlib],[System.Int32, mscorlib]][]
>> 109c320c     4726       170136
>> System.Linq.Expressions2.AssignmentExpression
>> 7912f348     7171       172104
>> System.Collections.Generic.List`1[[System.Reflection.MethodInfo,
>> mscorlib]]
>> 7911f538     1708       177632
>> System.Reflection.Emit.DynamicILGenerator
>> 7910ea30    15685       188220 System.RuntimeMethodHandle
>> 109c76cc     6304       201728 System.Linq.Expressions2.Block
>> 7912d9bc      889       204576 System.Collections.Hashtable+bucket[]
>> 790ff734    10742       214840 System.RuntimeType
>> 79130510    11613       223528 System.RuntimeTypeHandle[]
>> 791208b4    16407       262512 System.Reflection.Emit.GenericFieldInfo
>> 10f552cc    18594       297504
>> System.Collections.ObjectModel.ReadOnlyCollection`1[[System.Linq.Expres
>> sions2.Expression,
>> Microsoft.Scripting2.Core]]
>> 109c4f1c     9671       386840
>> System.Linq.Expressions2.MethodCallExpression
>> 7912dd40      646       432936 System.Char[]
>> 109c7634    16161       517152
>> System.Linq.Expressions2.ConstantExpression
>> 79109778    12835       718760 System.Reflection.RuntimeMethodInfo
>> 79135538     1494      1167624 System.Reflection.Emit.__FixupData[]
>> 7910dd38   124487      1493844 System.RuntimeTypeHandle
>> 7912d7c0    17031      1846316 System.Int32[]
>> 7912dae8    11218      1877012 System.Byte[]
>> 790fd8c4    46924      2987964 System.String
>> 7912d8f8   134460      5074708 System.Object[]
>>
>>
>> On Wed, Feb 25, 2009 at 10:51 PM, Dino Viehland <dinov at microsoft.com>
>> wrote:
>> > Your approach should be perfectly fine (it's basically what
>> CompiledCode was designed for) and it's not clear to me what would be
>> leaking in this scenario.
>> >
>> > One suggestion I'd have for you is to download and install windbg
>> (http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx).
>> >
>> > Then attach to your process which is using all the memory and type
>> in:
>> >
>> > .loadby sos mscorwks
>> > !DumpHeap -stat
>> >
>> > And send back the about the last 25 lines of that output.  That'll
>> give us the most common objects which are using memory in the GC heap
>> and we might be able to track it down from there.  If that's not
>> sufficient maybe we can iterate and figure out what's going on.
>> >
>> >> -----Original Message-----
>> >> From: users-bounces at lists.ironpython.com [mailto:users-
>> >> bounces at lists.ironpython.com] On Behalf Of Dody Gunawinata
>> >> Sent: Wednesday, February 25, 2009 8:38 AM
>> >> To: Dino Viehland
>> >> Cc: Discussion of IronPython
>> >> Subject: Re: [IronPython] Please vote up this DLR memory leak bug
>> >>
>> >> What is the impact of caching CompiledCode into this
>> "collectability"
>> >> issue?
>> >>
>> >> I am working on a CMS that expose functions implemented in
>> IronPython
>> >> to a template engine. So I read a directory of 40 or 50 xx.py files,
>> >> load it up and compiled them. This is off course a costly operation
>> to
>> >> do it in every single HTTP request (and SLOW) so I compiled the
>> source
>> >> code to CompiledCode and cache them.
>> >>
>> >> Then for every HTTP request, I retrieve these CompiledCode's from
>> the
>> >> Cache and call Execute(scope) to make the frozen code to be a real
>> >> function. So with X HTTP request, there will be X scope and  (X *
>> 40)
>> >> functions being generated at runtime.
>> >>
>> >> Now our memory consumption is huge and is getting bigger until IIS
>> >> gave up and recycle the process.
>> >>
>> >> Dody G.
>> >>
>> >>
>> >>
>> >>
>> >> > Just some comments on this as there's a few issues intertwined
>> here:
>> >> >
>> >> > 1.       Optimized code - this is currently the default and part
>> of
>> >> the
>> >> > original problem.  If you execute code in "optimized" mode we
>> >> generate an
>> >> > uncollectible type.  In 2.6 I'm switching the default to be non-
>> >> optimized
>> >> > and I also am trying out a change that should reduce the amount of
>> >> leakage
>> >> > when we do this.  That will make the out of the box experience
>> much
>> >> better.
>> >> >
>> >>
>> >>
>> >> > 2.       Multiple script runtimes - there's a bug in the DLR when
>> you
>> >> > repeatedly create script runtimes that the rules aren't getting
>> >> collected.
>> >> > This is supposed to be fixed in 2.6 but I'm still seeing some
>> leaks
>> >> in this
>> >> > scenario that I'll look into and fix.  Back-porting this to 2.0
>> would
>> >> be
>> >> > quite difficult as a lot of things have changed but we could
>> >> certainly look
>> >> > at backporting a different fix to 2.0.
>> >> >
>> >>
>> >> > 3.       There's a repro that defines "TestScriptOldClass" + n
>> which
>> >> leaks.
>> >> > All class / function / member names in code get interned and this
>> is
>> >> where
>> >> > this leak is coming from.  If you remove the + n then everything
>> >> works.  We
>> >> > may get rid of this interning at some point in the future but it's
>> a
>> >> fairly
>> >> > significant change.  I'm inclined to believe this one isn't as
>> much
>> >> of an
>> >> > issue as usually you aren't compiling code with new names every
>> >> single
>> >> > type.
>> >> >
>> >>
>> >>
>> >>
>> >>
>> >> >
>> >> >
>> >> > From: users-bounces at lists.ironpython.com
>> >> > [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dody
>> >> Gunawinata
>> >> > Sent: Sunday, February 22, 2009 2:10 PM
>> >> > To: Discussion of IronPython
>> >> > Subject: [IronPython] Please vote up this DLR memory leak bug
>> >> >
>> >> >
>> >> >
>> >> > There's only 5 votes so far.
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> http://www.codeplex.com/IronPython/WorkItem/View.aspx?WorkItemId=20399
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > nomadlife.org
>> >>
>> >>
>> >>
>> >> --
>> >> nomadlife.org
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users at lists.ironpython.com
>> >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>> >
>>
>>
>>
>> --
>> nomadlife.org
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>



-- 
nomadlife.org



More information about the Ironpython-users mailing list