[ python-Bugs-754449 ] Exceptions when a thread exits
SourceForge.net
noreply at sourceforge.net
Fri Jul 2 23:57:44 EDT 2004
Bugs item #754449, was opened at 2003-06-14 03:32
Message generated for change (Comment added) made by bcannon
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=754449&group_id=5470
Category: Threads
Group: Python 2.3
>Status: Closed
>Resolution: Fixed
Priority: 3
Submitted By: Matthias Klose (doko)
Assigned to: Brett Cannon (bcannon)
Summary: Exceptions when a thread exits
Initial Comment:
[forwarded from http://bugs.debian.org/195812]
The application mentioned is offlineimap, available
from ftp://ftp.debian.org/dists/unstable/main/source/.
This behavior is new to Python 2.3.
When my application exits, I get a lot of these messages:
Traceback (most recent call last):
File "/usr/lib/python2.3/threading.py", line 426, in
__bootstrap
self.__stop()
File "/usr/lib/python2.3/threading.py", line 435, in
__stop
self.__block.notifyAll()
File "/usr/lib/python2.3/threading.py", line 239, in
notifyAll
self.notify(len(self.__waiters))
File "/usr/lib/python2.3/threading.py", line 221, in
notify
currentThread() # for side-effect
TypeError: 'NoneType' object is not callable
jgoerzen at christoph:~/tree/offlineimap-3.99.18$
./offlineimap.py -l log -d maildir -a Personal,Excel
Unhandled exception in thread started by <bound method
ExitNotifyThread.__bootstrap of <ExitNotifyThread(Keep
alive LocalExcel, stopped daemon)>>
Traceback (most recent call last):
File "/usr/lib/python2.3/threading.py", line 426, in
__bootstrap
self.__stop()
File "/usr/lib/python2.3/threading.py", line 435, in
__stop
self.__block.notifyAll()
File "/usr/lib/python2.3/threading.py", line 239, in
notifyAll
self.notify(len(self.__waiters))
File "/usr/lib/python2.3/threading.py", line 221, in
notify
currentThread() # for side-effect
TypeError: 'NoneType' object is not callable
Unhandled exception in thread started by <bound method
ExitNotifyThread.__bootstrap of <ExitNotifyThread(Keep
alive RemoteExcel, stopped daemon)>>
Traceback (most recent call last):
File "/usr/lib/python2.3/threading.py", line 426, in
__bootstrap
self.__stop()
File "/usr/lib/python2.3/threading.py", line 435, in
__stop
self.__block.notifyAll()
File "/usr/lib/python2.3/threading.py", line 239, in
notifyAll
self.notify(len(self.__waiters))
File "/usr/lib/python2.3/threading.py", line 221, in
notify
currentThread() # for side-effect
TypeError: 'NoneType' object is not callable
----------------------------------------------------------------------
>Comment By: Brett Cannon (bcannon)
Date: 2004-07-02 20:57
Message:
Logged In: YES
user_id=357491
Fixed in 2.4 thanks to rev. 1.42 to Lib/threading. Don't know if this will
be backported to 2.3 . Possible, but need someone else to agree first.
----------------------------------------------------------------------
Comment By: k (carlosayam)
Date: 2004-06-02 23:05
Message:
Logged In: YES
user_id=990060
I don't understand one thing: how can python shut down
itself and, at the same time, keep a python code running in
a daemonised thread? or is that thread running in a
different "threaded" python byte-code interpreter?
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2004-06-02 22:25
Message:
Logged In: YES
user_id=31435
Brett, FYI, a daemon thread differs from a non-daemon
thread in only one respect: Python shuts down when only
daemon threads remain. It waits for non-daemon threads to
finish. So a daemon thread can keep running after the
interpreter has torn itself down completely. For that reason,
problems in daemon threads doing non-trivial things are
almost guaranteed.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-06-02 22:08
Message:
Logged In: YES
user_id=357491
Yep, that is how Python cleans up a module; sets everything in the
module to None and then removes it from sys.modules .
As for your case, I don't know enough about daemonized threads. My
patch for this only tries to let the exception make it to the top without
'threading' hiding the exception by causing its own exception from
interpreter shutdown. In other words I don't know if this is a related
issue or not.
----------------------------------------------------------------------
Comment By: k (carlosayam)
Date: 2004-06-02 20:01
Message:
Logged In: YES
user_id=990060
bcannon said on 2004-02-17
>> Does anyone know how to cause this error in isolation?
I'm getting the same error when I start a new thread,
daemonise it and the thread goes into a very slow database
operation (loading a large object); meanwhile the main
thread starts a GUI; then I close the window, exiting the
main loop and the python interpreter ends (or tries to end.)
In relation to bcannon comment on how to reproduce the error
(setting all variables to None in the module), my guess is
that while exiting, the python interpreter is somehow
freeing all variables in the module (cleaning the module or
something), but the module is still running and that raises
the error... is this possible?
Note: if the thread is not daemonised, the problem desapears
but the script (the python interpreter) takes a while to finish.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-05-16 21:36
Message:
Logged In: YES
user_id=357491
How to reproduce the bug:
* Follow the instructions at http://mnetproject.org/repos/ on how to get a
copy of the in-dev version of mnet (I used the instructions for using wget
and it worked fine).
* Run ``python setup.py``. Do realize that this app will download other
Python code (crypto stuff mostly) to finish its install. Takes up about 54
MB on my machine after it is compiled.
* Run ``python setup.py test -a``. This executes the testing suite. The
bug manifests itself after the testing suite finishes execution.
This will consistently cause the bug every time.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-05-16 11:45
Message:
Logged In: YES
user_id=357491
Discovered this is not fixed after all (previous fix didn't go far enough;
still needed, though). Patch 954922 is the second attempt to fix this.
This time, though, I had code that could trigger the problem reliably and
thus this should be the proper, and final, fix.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-03-08 14:36
Message:
Logged In: YES
user_id=357491
OK, this has been fixed in Python 2.4 (rev. 1.41 of Lib/threading.py; just
removed the two calls to currentThread in the _Condition class that were
commented to be for their side-effect).
Just because I don't feel rock-solid on my fix to the point of it not
changing some obscure semantics that I would be willing to put a gun to
my head over it I am not going to backport this until I hear from
someone who had this bug and reports that they have not had any issues
for a long time. Otherwise I will just leave the fix in 2.4 only.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-03-08 14:32
Message:
Logged In: YES
user_id=357491
OK, this has been fixed in Python 2.4 (rev. 1.41 of Lib/threading.py; just
removed the two calls to currentThread in the _Condition class that were
commented to be for their side-effect).
Just because I don't feel rock-solid on my fix to the point of it not
changing some obscure semantics that I would be willing to put a gun to
my head over it I am not going to backport this until I hear from
someone who had this bug and reports that they have not had any issues
for a long time. Otherwise I will just leave the fix in 2.4 only.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-03-02 20:29
Message:
Logged In: YES
user_id=357491
To force this error I inserted code to set all attributes of the threading
module to None and then throw and exception. After commenting out
the two calls to currentThread in _Condition the thrown exception to
trigger the problem propogated to the top correctly.
I did have the clean-up function give out since I set *everything* to
None, but it doesn't look like a normal issue as shown in the exception
traceback in the OP. I don't think it is an issue.
So it seems commenting the two calls to currentThread in _Condition
solves the problem. But since this is threading code and the comment
explicitly states it is for the side-effect I am going to try to get a second
opinion.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-02-16 20:59
Message:
Logged In: YES
user_id=357491
Does anyone know how to cause this error in isolation? I have tried a
bunch of different ways but cannot cause an exception to be raised at the
correct point in Threading.__bootstrap() to lead to self.__stop() to be
called while the interpreter is tearing itself down.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2004-02-09 17:18
Message:
Logged In: YES
user_id=357491
After staring at the code, I am not even sure if the calls for its side-
effect are needed. If you call currentThread(), it either returns a value
from _active which is a dict of running Thread instances indexed by
thread idents, or a new _DummyThread instance that inserts itself into
_active. Now the two calls in the _Condition class are purely for this
possible side-effect.
And yet at no point is anything from _active, through currentThread or
direct access, even touched by a _Condition method. The only chance
this might be an issue is if a _Condition instance uses an RLock instance
for its locking, which is the default behavior.
But this still makes the side-effect need useless. RLocks will call
currentThread on their own and since this is all in the same thread the
result won't change. And in this specific case of this bug, the _Condition
instance is created with a Lock instance since that is what the Thread
instance uses for constructing the _Condition instance it use, which is just
thread.allocate_lock() which is obviously not an RLock.
In other words I can't find the point to the side-effect in _Condition. I
will try to come up with some testing code that reproduces the error and
then see if just removing the calls in _Condition remove the error and
still pass the regression tests.
----------------------------------------------------------------------
Comment By: John Goerzen (jgoerzen)
Date: 2003-06-16 06:26
Message:
Logged In: YES
user_id=491567
I can confirm that this behavior is not present in Python 2.2 in
the same version that I am using to test against Python 2.3.
I will be on vacation for most of this and next week. I'll try to
get to the logging script before I leave, but I might not get to it
until I return.
FYI, you can also obtain OfflineIMAP at
http://quux.org/devel/offlineimap.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2003-06-15 23:58
Message:
Logged In: YES
user_id=357491
OK, following Tim's advice I checked and it seems that Thread
calls a method while shutting itself down that calls
Condition.notifyAll which calls currentThread which is a global. It
would appear that offlineimap is leaving its threads running, the
program gets shut down, the threads raise an error while
shutting down (probably because things are being torn down),
this triggers the stopping method in Thread, and this raises its
own exception because of the teardown which is what we are
seeing as the TypeError.
So the question is whether Condition should store a local
reference to currentThread or not. It is not the most pure
solution since this shutdown issue is not Condition's, but the only
other solution I can think of is to have Thread keep a reference
to currentThread, inject it into the current frame's global
namespace, call Condition.notifyAll, and then remove the
reference from the frame again. I vote for the cleaner, less pure
solution. =)
Am I insane on this or does it at least sound like this is the
problem and proper solution?
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2003-06-15 23:19
Message:
Logged In: YES
user_id=357491
Nuts. For some reason I thought the OP had said when threads
were exiting.
I will ask on python-dev for a good explanation of what happens
when Python is shutting down.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2003-06-15 18:05
Message:
Logged In: YES
user_id=31435
Note that the OP said "when my application exits". When
Python is tearing down the universe, it systematically sets
module-global bindings to None. currentThread() is a module
global. I can't make more time for this now, so for more info
talk about it on Python-Dev.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2003-06-15 16:44
Message:
Logged In: YES
user_id=357491
Well, I'm stumped. I checked the diff from when 2.2 was initially
released until now and the only change that seems to be related
to any of this is that what is returned by currentThread is not
saved in a variable. But since the error is the calling of
currentThread itself and not saving the return value I don't see
how that would affect anything.
I also went through offlineimap but I didn't see anything there
that seemed to be setting currentThread to None. Although
since several files do ``import *`` so there still is a possibility of
overwriting currentThread locally.
So, for my owning learning and to help solve this, I have written
a tracing function that writes to stderr using the logging package
when it detects that either currentThread or
threading.currentThread has been set to None, locally or globally
(I assume the code is not injecting into builtins so I didn't bother
checking there). The file is named tracer.py and I have attached
it to this bug report. If you can execute
``sys.settrace(tracer.trace_currentThread)`` before offlinemap
starts executing and immediately within each thread (it has to be
called in *every* thread since tracing functions are no inherited
from the main thread) it should print out a message when
currentThread becomes None. If you *really* want to make this
robust you can also have it check sys.modules['threading'] every
time as well, but I figure there is not going to be much renaming
and masking of currentThread.
----------------------------------------------------------------------
Comment By: Matthias Klose (doko)
Date: 2003-06-15 01:44
Message:
Logged In: YES
user_id=60903
Please see
http://packages.debian.org/unstable/mail/offlineimap.html
or for the tarball:
http://ftp.debian.org/debian/pool/main/o/offlineimap/offlineimap_3.99.18.tar.gz
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2003-06-15 01:19
Message:
Logged In: YES
user_id=357491
I went to the FTP site and all I found was some huge,
compressed files (after changing the path to ftp://ftp.debian.org/
debian/dists/sid/main/source); no specific program called
offlinemap. If it is in one of those files can you just add the file
to the bug report?
As for the reported bug, it looks like currentThread is being
redefined, although how that is happening is beyond me. I
checked the 'threading' module and no where is currentThread
redefined which could lead to None. Has the app being run been
changed at all since Python 2.2 was released?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=754449&group_id=5470
More information about the Python-bugs-list
mailing list