[ python-Bugs-1089632 ] _DummyThread() objects not freed from threading._active map

[SourceForge.net]

Bugs item #1089632, was opened at 2004-12-22 02:07
Message generated for change (Comment added) made by bcannon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1089632&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
>Status: Closed
Resolution: Wont Fix
Priority: 5
Submitted By: saravanand (saravanand)
Assigned to: Nobody/Anonymous (nobody)
Summary: _DummyThread() objects not freed from threading._active map

Initial Comment:
Problem Background:
===============

I have Python Server module (long running) which 
accepts calls from several Python Clients over socket 
interface and forwards the call to a C++ component.  
This C++ component gives the reponses back to Python 
Server in a separate thread(created by C++ module) via 
callback.

In the Python Callback implementation, the responses 
are sent to client in a synchronised manner using Python 
primitive threading.Semaphore.  This Synchronisation is 
required as the C++ component can deliver parallel 
responses in different C++ threads.

Here, the Python Server creates the semaphore object 
per client when the client request arrives (in Python 
thread).  This same object is acquired & released in the 
C++ callback thread(s).

Here we observed that Windows Events are getting 
created whenever the acquire method is executed in the 
Python Callback implementation in the context of C++ 
thread. But the same event is not freed by the Python 
Interpreter even after the termination of the C++ 
thread.   Because of this, a Windows Event handles are 
getting leaked in the Python Server.  

Problem Description:
==============
When we checked the Python module threading.py, we 
found that, every time a non-python thread (in our case 
C++ created thread), enters python and accessesn a 
primitive in threading module (eg: Semaphore, RLock), 
python looks for an entry for this thread in the _active 
map using thread ID as the Key. Since no  entry exists 
for such C++ created threads, a _DummyThread object 
is created and added to the _active map for this C++ 
thread. 

For every _DummyThread object that is created, there is 
a corresponding Windows Event also getting created.

Since this entry is never removed from the _active map 
even after the termination of the C++ thread ( as we 
could make out from the code in threading.py),for 
every "unique" C++ thread that enters python, a 
Windows Event is allocated and this manifests as 
continuous increase in the Handle count in my Python 
server ( as seen in Windows PerfMon/Task Manager).

Is there a way to avoid this caching in Python 
Interpreter? Why cant Python remove this entry from 
the map when the C++ thread terminates. Or if Python 
can't get to know about the thread termination, should 
it not implement some kind of Garbage collection for the 
entries in this Map (especially entries for the 
_DummyThread objects).

Does this require a correction in Python 
modulethreading.py?

or is this caching behaviour by design?

----------------------------------------------------------------------

>Comment By: Brett Cannon (bcannon)
Date: 2005-01-05 23:19

Message:
Logged In: YES 
user_id=357491

What does Semaphore have to do with _DummyThread and 
currentThread?

And as for win32api, that is not part of the Python stdlib and thus I have 
no way of commenting on that.

For C++ threads to call into Python code, I would not say that is bad 
specifically.  You were just trying to get thread info for non-Python 
threads and that was leading to _DummyThread instances to be created 
as designed and you just didn't want that.  Calling Python code is fine, 
just don't expect it to know about your C++ threads as much as you 
seem to want it to.

And please leave this bug closed.  If free to submit a patch to change 
semantics, but that does not affect this bug report.

----------------------------------------------------------------------

Comment By: saravanand (saravanand)
Date: 2005-01-05 02:54

Message:
Logged In: YES 
user_id=1181691

I tried the following workaround which is working (causes no 
handle leaks)

Workaround is to change threading.semaphore to Windows 
Extension module APIs win32event.CreateMutex(),  
win32event.WaitForSingleObject and win32event.ReleaseMutex
()

After this change, there are no handle leaks. So, my 
questions are:
1) Is this workaround OK or are there any other issues 
regarding the win32api usage ?
2) you suggested to create minimal python wrappers for C++ 
code and call C++ from python (instead of C++ thread 
callbacks). So I would like to know,  in general, whether it is a 
bad idea for c++ threads to callback into Python. If yes, what 
are the issues (apart from the handle leak mentioned before). 
If no, I would like to live with the above workaround.

Thanks in advance

----------------------------------------------------------------------

Comment By: saravanand (saravanand)
Date: 2005-01-05 02:29

Message:
Logged In: YES 
user_id=1181691

asdfas

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2004-12-23 18:35

Message:
Logged In: YES 
user_id=357491

Yes, it is by design.  If you read the source you will notice that the 
comment mentions that the _DummyThread object is flagged as a 
daemon thread and thus should not be expected to be killed.  The 
comment also mentions how they are not garbage collected.  As stated in 
the docs, dummy threads are of limited functionality.

You could cheat and remove the entries yourself from threading._active, 
but that might not be future-safe.  I would just make sure that all 
threads are created through the threading or thread module, even if it 
means creating a minimal wrapper in Python for your C++ code to call 
through that to execute your C++ threads.

If you want the docs to be more specific please feel free to submit a 
patch for the docs.  Or if you can come up with a good way for the 
dummy threads to clean up after themselves then you can also submit 
that.

But since the source code specifies that this expected and the docs say 
that dummy threads are of limited functionality I am closing as "won't 
fix".

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1089632&group_id=5470