[IronPython] clr.AddReference effect how GC work ?!?!?!?

[Dino Viehland]

Yikes, this is a scary bug...  This is related to COM and STA threads even though you're not using COM!  I might get some of this analysis wrong I think I understand basically what's happening.

The main thread of all Python apps is an STA thread.  This enables things like WinForms to work from a .py script.  It also enables other things like starting processes which use shell execute - all shell APIs require to be called on an STA thread.  We used to have an option, and it will be coming back, -X:MTA which would force an app to start it's "main thread" in MTA mode.

It looks like the CLR is remembering that some object it has created were in fact created on an STA thread and therefore the finalizer needs to run on the STA thread as well.  So the CLR finalizer thread is blocked trying to get back to the STA thread.  Meanwhile you're sitting in a loop which is doing GC collections and a Thread.Sleep.  This prevents the CLR from ever getting back to the STA thread because it is not pumping messages.

So obviously one impractical solution is to pass the non-existent -X:MTA option :).  Another solution is to ensure that you pump messages.  The easiest way to do that in your sample code below is Thread.CurrentThread.Join(100) instead of doing a Thread.Sleep(100).  The join will continue to pump messages while waiting for the current thread to exit.  That will time out of course because it's waiting on its self so it'll behave just like a sleep.

Anyway I've opened a bug to track this: http://www.codeplex.com/IronPython/WorkItem/View.aspx?WorkItemId=16249  We at the very least need to bring back -X:MTA.  You could also imagine us doing something useful like warning when you call Thread.Sleep - but first I need to implement warning support :).  Finally we could see if we could avoid creating the object which has to be finalizable on an STA thread but that might be more of a challenge - I'm not even sure what object it is.

-----Original Message-----
From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Sakesun Roykiattisak
Sent: Wednesday, April 23, 2008 12:23 AM
To: users at lists.ironpython.com
Subject: [IronPython] clr.AddReference effect how GC work ?!?!?!?


Hi,

     My today attempt to fight with over-memory usage in my IronPython server application, reveal a very strange behavior of IronPython (both 1.1 and 2.0b)  I'll demonstrate with the following script:
"loopit.py"


##### loopit.py ###########################

import clr
#clr.AddReference('System')

class MyObj(object):
     def __del__(self): print 'gone'

def run():
     from System.Threading import Thread
     from System import GC
     while True:
         MyObj()
         Thread.Sleep(100)
         GC.Collect()

run()

###########################################


try to run this script you'll see a stream of 'gone' print out.
However, if you uncomment the line "clr.AddReference('System')"
and rerun it again the system will simple halt, memory usage will keep raising. Until you press Ctrl-C, then the you will see the pending 'gone' stream out.

I've tried with

clr.AddReference('System.Data')
clr.AddReference('System.Xml')

and the behaviour is the same for any Assembly.
This is really weird. Why should clr.AddReference effect how GC work ?

And I believe this is indeed the true source of memory problem I'm facing.

Hope someone explain me why.

Thanks