[Python-bugs-list] [ python-Bugs-704180 ] Memory leak: threading.Thread.__init__(self, target=self.xx)

SourceForge.net noreply@sourceforge.net
Sun, 23 Mar 2003 05:52:26 -0800


Bugs item #704180, was opened at 2003-03-15 16:44
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=704180&group_id=5470

Category: Threads
Group: Python 2.2.2
Status: Closed
Resolution: Invalid
Priority: 5
Submitted By: Darrell Gallion (dgallion)
Assigned to: Nobody/Anonymous (nobody)
Summary: Memory leak: threading.Thread.__init__(self, target=self.xx)

Initial Comment:
This call  
	threading.Thread.__init__(self, target=self.xxx) 
makes this assignment  
	self.__target = target 
Which looks like a circular refrence to me.  
weakref anyone ? 
 
The following test code shows the leak in action. 
 
--Darrell 
 
 
import threading, time 
 
class T1(threading.Thread): 
    def __init__(self, leak=0): 
        self.leak=leak 
        if leak: 
            threading.Thread.__init__(self, target=self.m1) 
        else: 
            threading.Thread.__init__(self) 
        self.start() 
         
    def run(self): 
        pass 
         
    def m1(self): 
        pass 
         
    def __del__(self): 
        print 'DEL:', self, "leak:",self.leak 
         
 
for x in range(10): 
    T1(leak=1) 
for x in range(10): 
    T1() 
time.sleep(1) 
 
 

----------------------------------------------------------------------

>Comment By: Darrell Gallion (dgallion)
Date: 2003-03-23 13:52

Message:
Logged In: YES 
user_id=68151

My problem was solved by moving the thread options into a group of 
functions and passing one of those as the target. These threads have a 
shutDown method that didn't work before because of this target problem.  
 
This isn't a problem for me now, yet it seemed like it could have been 
avoided in the first place.  
 
Thanks 

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-03-22 22:45

Message:
Logged In: YES 
user_id=31435

As Martin said, a weakref won't solve anything here.

I remain unclear on what the real problem is.  You said that 
the timing of when an object gets destroyed is important to 
your app, but you have to realize that Python makes no 
guarantees about object destruction time.  If your thread is 
holding on to some critical resource that must be freed in a 
timely manner, the only sane way to proceed is to give your 
thread a release() or close() method that's called explicitly 
to free the critical resource.  That should never be left to 
the vagaries of garbage collection (whether via refcount or 
the gc module).

BTW, I don't believe I've ever seen code that passed a 
bound method of a Thread subclass to Thread.__init__ 
before.  The target argument was meant to ease migrating 
old code that used the thread module (where specifying a 
target function was necessary).  Code written with 
threading.py in mind invariably (IME) overrides the run() 
method instead (which is the intent of the design).

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-03-22 21:30

Message:
Logged In: YES 
user_id=21627

That won't work. In your example, the method object is
created with the expression self.m1, and the only reference
to it is the target parameter. So if Thread would keep just
a weak reference to target, then the method object would go
away just after __init__ completes. As the result, the
Thread's .run won't find the target.


----------------------------------------------------------------------

Comment By: Darrell Gallion (dgallion)
Date: 2003-03-22 20:36

Message:
Logged In: YES 
user_id=68151

Using a threading.Thread as a base class invites you to pass target methods 
other than run. Which sets up the cycle. It would be nice if the target could 
be a weakref or was converted to a weakref by Thread. 
 
This simple change would prevent others from wasting time on this problem. 
 

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-03-21 18:22

Message:
Logged In: YES 
user_id=21627

I fail to see a bug in Python here. You are passing a
reference to an object to the __init__ of the base class, so
you should know you are creating cycles.

I recommend to re-implement T1.run instead of using the
target= constructor argument.

----------------------------------------------------------------------

Comment By: Darrell Gallion (dgallion)
Date: 2003-03-16 16:45

Message:
Logged In: YES 
user_id=68151

I know about the __del__ issue with gc, and your right this example worked 
fine with it removed. The much larger app that sent me on this chase wasn't 
fixed by the __del__ method. Yet it was fixed by moving the target method 
to a function call instead of a method on the object. 
 
It's a large app and the timing of when an object is destroyed matters. 
The following helps me see my problem.  
 
import threading, time, weakref 
 
class T1(threading.Thread): 
    def __init__(self): 
        self.go=1 
        threading.Thread.__init__(self, target=self.m1) 
        self.start() 
         
    def m1(self): 
        while self.go: 
            time.sleep(0.0001) 
         
 
objs={} 
while 1: 
    t1 = weakref.ref(T1()) 
    objs[t1]=1 
    t1().go=0 
    time.sleep(0.01) 
    for o in objs.keys(): 
        if o() == None: 
            del objs[o] 
    print len(objs), 
 
 
         
 
 
 

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-03-16 04:39

Message:
Logged In: YES 
user_id=31435

The "leak" is due to that you gave this class a __del__ 
method:  cyclic garbage collection won't magically reclaim 
objects in cycles with __del__ methods.  Remove the 
__del__ and you can run this all day without the process 
size growing.  If you leave the __del__ in, you can find all 
your uncollectible thread objects in gc.garbage (and use 
that to break the cycles and get them collected, if you like --
 I'd recommend not using __del__ at all, though).

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=704180&group_id=5470