[IronPython] Jit leak in defaultdict for Ipy 2.6.2

[Dino Viehland]

The closest non-windbg solution I could think of might be CLR Profiler which can probably report the number of dynamic method objects which are alive.  Windbg can also do this when you dump the heap and is probably over all actually easier to use in this case as you can attach, do !DumpHeap -stat, and then detach.

Just one more data point which might help you understand what's going on - the JITed methods in this case are probably adding 4k of memory to the cost of each defaultdict instance as long as the defaultdict is alive.

From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg
Sent: Monday, November 29, 2010 9:59 AM
To: Discussion of IronPython
Subject: Re: [IronPython] Jit leak in defaultdict for Ipy 2.6.2

Thanks.
How I suspect there is a leak in the JIT of my application, up until now I thought that if the performance counter for "# of methods jitted" is constantly rising then that means exactly that. From your reply I understand that this is not the case.
Can you tell how can I know how many uncollectible JIT objects I have so that I can trace them (preferably without using windbg)

Thanks again...

From: users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com> [mailto:users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com>] On Behalf Of Dino Viehland
Sent: Monday, November 29, 2010 7:41 PM
To: Discussion of IronPython
Subject: Re: [IronPython] Jit leak in defaultdict for Ipy 2.6.2

Defaultdict is creating a new invoke binder - it should be getting the binder from PythonContext using the Invoke(CallSignature) method.  Because it creates a new binder each time we are getting no caching of the rules across defaultdict instances and it'll end up generating a new method to handle the missing call.  It is collectible (so not really a leak) but it is really bad from a performance perspective.

From: users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com> [mailto:users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com>] On Behalf Of Idan Zaltzberg
Sent: Monday, November 29, 2010 4:10 AM
To: Discussion of IronPython
Subject: [IronPython] Jit leak in defaultdict for Ipy 2.6.2

Hi,
I have noticed the following method always adds a jitted method (looking at the ".NET CLR Jit" performance counter) when it is run:
def f():
                d = defaultdict(int)
                d[0]

I created my own implementation of defaultdict (in ipy):
class defaultdict(dict):
    def __init__(self, cls):
        super(defaultdict, self).__init__()
        self.cls = cls
    def __getitem__(self, key):
        if key not in self:
            self[key] = self.cls()
        return super(defaultdict, self).__getitem__(key)

And I noticed that it does not leak JIT and it works 200 times faster when running the method f().
Can you please look why this happens in the current implementation?
Also I was wondering if there are any other utility methods that use similar code and probably will have the same problem.

Thanks,
Idan zalzberg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20101129/536eff95/attachment.html>