[Ironpython-users] Memory Leak and Lightweight Scopes

[Dino Viehland]

Marcus wrote:
> Hi,
> 
> We currently have a massive memory leak using IronPython hosted in our
> application (about 10-15 MB per script run).
> 
> Googling brought up that using the LightweightScopes option should solve
> that problem. However, I was not able to find any documentation about the
> implications of lightweight scopes.

We have different modes on how we can compile the top-level code.  Originally
one of those modes was to generate types with static fields and generate code into 
the static types.  That let us quickly access relevant data (it's in a static) but would
make it slower to generate the code (RefEmit is slower than a dynamic method) and
it also made the code uncollectible.

We changed this at some point to improve startup by using a set of fields which
are baked into IronPython.dll and we hand out and use via DynamicMethods instead of
doing RefEmit - which makes it a little bit easier to figure out if this is the cause of your
leak.

To see if this is the source of your leak you could look at the following classes in the
IronPython.Compiler.Ast namespace:

StorageData
ContextStorage00*
GlobalStorage00*
SiteStorage*<T>

If the fields in those classes aren't getting populated then this is not the cause of the leak.

I'm actually having trouble finding the exact line of code where we make the decision on
which mode to use, but passing -X:LightweightScopes should definitely still work to flip
this mode off.  I also think we're less aggressive about enabling it now and that in general
only command line apps should see this enabled by default.  Also make sure that you don't 
have debugging enabled as that forces to create uncollectible code too and probably 
overrides the light weight scopes option.

> Additionally, it seems that this option does not solve our problem, at least
> not fully. (I'm still researching the details.)
> 
> Does anyone have any hints where to redirect my research?

My favorite technique is to just use WinDBG + sos to dump the heap and then diff the 
heaps.  Usually after enough runs it gets to be pretty obvious.  Just download windbg,
attach to the process, and do:

.loadby sos clr
!DumpHeap

Continue execution, break, dump the heap again.  From there you'll usually start to see
Certain objects growing in numbers, and then you can do:

!GCRoot <addr>

And it'll figure out who's keeping the object alive.