[Python-bugs-list] [ python-Bugs-215076 ] threads of __del__
noreply@sourceforge.net
noreply@sourceforge.net
Thu, 09 Aug 2001 10:04:18 -0700
Bugs item #215076, was opened at 2000-09-22 02:15
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=215076&group_id=5470
Category: Threads
Group: None
>Status: Closed
>Resolution: Wont Fix
Priority: 3
Submitted By: Toby Dickenson (htrd)
Assigned to: Martin v. Löwis (loewis)
Summary: threads of __del__
Initial Comment:
I suspect (but have not proved) that the support for destroying deeply recursive structures can allow for an instances __del__ method to get called in a thread other than the one in which it lost its last reference.
This can happen if it switches to a different thread during deletion of a deeply recursive structure (with _PyTrash_delete_later not empty).
Should Python guarantee that this does not happen? (I say yes - it is important for COM clients)
----------------------------------------------------------------------
>Comment By: Martin v. Löwis (loewis)
Date: 2001-08-09 10:04
Message:
Logged In: YES
user_id=21627
Since the patch that was supposed to fix this problem got
rejected, it appears that this problem (whatever the problem
exactly is) won't get fixed. It appears that this report is
asking for an enhancement of Python, rather than reporting a
bug. If you still think something needs to be done, I
recommend to produce a PEP explaining the problem and the
proposed solution in detail.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-08-09 09:04
Message:
Logged In: YES
user_id=31435
Assigning to Martin, since the corresponding patch got
assigned to him too.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-03-22 18:28
Message:
Logged In: NO
Im trying to learn the art of cracking
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2000-09-25 02:04
Message:
Ive noticed that this patch also fixes a different problem that I hadnt realised was related....
We were observing that one thread occasionally became blocked for a few tens of seconds, and the profiler wasnt giving any hints.
It turns out that this thread was busy emptying the trashcan, and running the deallocator functions for the objects used and discardered by all the other threads.
The script below shows up this problem (on NT). It prints pairs of numbers; the first out of each pair will stay the same for 'a long time'.
Mark: I assume Python.net will print the 'Waaah' lines too?
import thread,time
class A:
def __init__(self,other):
self._other = other
self._ident = thread.get_ident()
def __del__(self):
new_ident = thread.get_ident()
if new_ident!=self._ident:
print 'Waaaaaah', new_ident, self._ident
def work(size):
ident = thread.get_ident()
while 1:
a=None
for i in range(size):
a = A(a)
for i in range(10):
a = (a,)
# Three few big ones
for i in range(3):
thread.start_new_thread(work,(1000,))
# And one huge one
work(100000)
----------------------------------------------------------------------
Comment By: Mark Hammond (mhammond)
Date: 2000-09-22 16:57
Message:
I've got to side with Tim that "last thread with a reference" is not the correct semantic to be applied here.
On the other hand, I do agree that some determinism would be nice here. The reporter is completely correct about, eg, COM rules. If I was BDFL, I would probably go for something deterministic, and that probably is "last thread to release", as anything else is just unreasonable.
My understanding of .NET is that it does GC on a seperate GC thread! So there you are guaranteed to not get the behaviour (but guaranteed not to get it ever ;-)
Sorry Tim - back at ya!
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2000-09-22 11:52
Message:
Assigned to MarkH, as I want his opinion on this.
I dislike it on principle: it elevates an accident of refcount semantics into a design goal, and complicates the code to achieve that.
The Microsoft .NET implementation of Python doesn't do refcounting, so not even the pre-trashcan refcount semantics can be relied on there, and I don't want to add to users' difficulties in moving among Python implementations (despite that it may be in my immediate financial interest to lock people into CPython, I'll leave "embrace & extend" to MS <wink>).
If this sounds like a good idea to Mark anyway, fine, but I don't buy it otherwise.
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2000-09-22 07:52
Message:
There is a patch to move the trashcan into the thread state. see http://sourceforge.net/patch/?func=detailpatch&patch_id=101606&group_id=5470
This uses PyThreadState_GET() once to avoid the extra function call in the normal path, and only uses the trashcan if the thread state is non NULL. (assuming we are never deleting deeply recursive objects with no thread state)
On performance, this gives me a 2% increase in pystones....
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2000-09-22 06:30
Message:
It sounds that it will be hard to enforce that __del__ is called in the constructing thread even without the complication of the trashcan code.
Explicitly referencing thread-specific data in a __del__ seems silly -- if the object is tied so closely to the thread, why not add a pointer to the thread to the object.
Nevertheless, especially the COM thing sounds like it would be a good thing to prevent this if it is possible, and your solution (make the trashcan state part of the thread state) sounds reasonable -- except that it means that every deallocation of a tuple, list, dict, frame or traceback needs yet another call to PyThreadState_Get(). And, most disturbing of all, we found before releasing 2.0b1 that there are situations where dictionaries (and possibly other objects) are used when there is no current thread state!!!
Replacing one theoretical problem with another not-so-theoretical one doesn't seem right. Perhaps we could invent a flag that prevents thread-switches while the trashcan dealloc code is running? This could be a simple global boolean that, when set, stops the ceval main loop (and the BEGIN/END ALLOW THREADS macros and related functions!) from ever yielding the interpreter loop. It would almost never be triggered in practice, but guarantee safety in worst cases -- if not efficiency (other threads would be completely blocked until the trashcan code is done).
Back to Tim...
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2000-09-22 03:27
Message:
There are two cases that I have in mind:
First is that of a COM client.
It depends on the COM threading model in use, but this problem applies in the most common threading model which is also the default threading model using Mark Hammond's pythoncom.
It is only permitted to access a reference to a COM object from the thread in which it is created. This restiction applies to the Release() method that is used by pythoncom to release its reference to the COM server.
Secondly, it is common to use a dictionary keyed by thread.get_ident() to provide thread-local storage. This is used extensively in Zope. Any application that uses this type of thread-local storage in a __del__ is likely to be broken.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2000-09-22 03:08
Message:
And why, exactly, would it be a problem that the trash is deleted in the 'wrong" thread? It's just trash. Any thread can delete it.
Or am I missing something?
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2000-09-22 03:05
Message:
I agree there is no self-consistency problem with the trashcan, however......
The code in _PyTrash_destroy_chain can call an objects __del__ method, which can release the interpreter lock.
A second thread may then reenter _PyTrash_destroy_chain and continue to clean up the first thread's garbage. (the first thread is safely blocked inside the Py_DECREF)
Yes the garbage gets cleaned up, however it happens in the wrong thread.
I suggest that _PyTrash_delete_later should be part of pyThreadState
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2000-09-22 02:52
Message:
Agreed that this would be a problem if it could cause an inconsistent state of the trashcan (at the end of object.c) structure. However, since the "trash" code is all executed with the global interpreter lock held, I don't see that there can be a problem. Here's why:
The code in _PyTrash_deposit_object() doesn't make any calls that could invoke __del__, so it's safe. The code in _PyTrash_destroy_chain() makes exactly one call to Py_DECREF(), but it takes care that its global data structures are consistent at this point. For a second I thought that _PyTrash_delete_later would have to be declared volatile, but on second thought I don't: the Py_DECREF() macro expands to a possible (indirect) function call, signaling the compiler that the global _PyTrash_delete_later may be changed in arbitrary ways.
Assigned to Tim Peters to verify my reasoning, because of his superior mind in reasoning about threads and race conditions.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=215076&group_id=5470