Threading problems at program exit

Dave Cole djc at object-craft.com.au
Sat Nov 30 17:36:07 EST 2002


Dave> The following program demonstrates a problem I am experiencing with
Dave> the threading module.  I have an object (A) which holds a lock on
Dave> another object (B).  When object A is deleted I want it to release any
Dave> lock it may still be holding on object B.

Dave> Everything works fine except when the program terminates.

Dave> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dave> import sys
Dave> import threading

Dave> class Locked:
Dave>     def __init__(self, lock):
Dave>         self._lock = lock
Dave>         self._lock_count = 0
Dave>         self._thread = None
Dave>         self._log = sys.stderr.write
Dave>         self._current_thread = threading.currentThread
Dave>         self.lock()

Dave>     def lock(self):
Dave>         self._log('locked in %s\n' % self._current_thread())
Dave>         self._lock.acquire()
Dave>         self._lock_count += 1

Dave>     def unlock(self):
Dave>         self._log('unlocked in %s\n' % self._current_thread())
Dave>         self._lock_count -= 1
Dave>         self._lock.release()

Dave>     def __del__(self):
Dave>         while self._lock_count:
Dave>             self.unlock()

Dave> obj = Locked(threading.RLock())
Dave> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Dave> When I run this program I get the following:

Dave> locked in <_MainThread(MainThread, started)>
Dave> unlocked in <_DummyThread(Dummy-1, started daemon)>
Dave> Exception exceptions.AssertionError: <exceptions.AssertionError instance at 0x814fbfc> in <bound method Locked.__del__ of <__main__.Locked instance at 0x816e5f4>> ignored

Dave> It looks like interpreter is deleting thread objects before objects
Dave> which hold locks in those threads.  Is there any kosher way I can
Dave> avoid the problem?

Dave> The only way I can think to fix it is a bit non-kosher:

Dave> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dave> import sys
Dave> import threading

Dave> class Locked:
Dave>     def __init__(self, lock):
Dave>         self._lock = lock
Dave>         self._lock_count = 0
Dave>         self._thread = None
Dave>         self._log = sys.stderr.write
Dave>         self._current_thread = threading.currentThread
Dave>         self.lock()

Dave>     def lock(self):
Dave>         self._log('locked in %s\n' % self._current_thread())
Dave>         self._lock.acquire()
Dave>         self._lock_count += 1

Dave>     def unlock(self):
Dave>         self._log('unlocked in %s\n' % self._current_thread())
Dave>         self._lock_count -= 1
Dave>         self._lock.release()

Dave>     def __del__(self):
Dave>         if self._lock_count:
Dave>             count, owner = self._lock._release_save()
Dave>             self._log('owner was %s\n' % owner)
Dave>             owner = self._current_thread()
Dave>             self._log('owner is now %s\n' % owner)
Dave>             self._lock._acquire_restore((count, owner))
Dave>             while self._lock_count:
Dave>                 self.unlock()

Dave> obj = Locked(threading.RLock())
Dave> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Dave> This prints the following:

Dave> locked in <_MainThread(MainThread, started)>
Dave> owner was <_MainThread(MainThread, stopped)>
Dave> owner is now <_DummyThread(Dummy-1, started daemon)>
Dave> unlocked in <_DummyThread(Dummy-1, started daemon)>

Dave> The trouble is that it requires the use of private methods of the
Dave> RLock class.

Dave> Is there a better or more kosher way?

Ype> You might consider not using the __del__ method at all because
Ype> 'it is not guaranteed that __del__() methods are called for
Ype> objects that still exist when the interpreter exits. ' (quoted
Ype> from the language ref).  Then use:
Ype> 
Ype> obj = Locked(threading.RLock())
Ype> try:
Ype>      # whatever needs to be done with obj
Ype> finally:
Ype>     obj.unlock()

That won't work in this case because the resource needs to be locked
over a span of client code.

I should be more specific...

The code in question is my Python bindings for Sybase.  According the
the DB-API specification a database module must provide Connection
objects for managing database connections, and Cursor objects for
executing commands over a Connection.  The DB-API implies that you
should be able to do something like the following:

db = Sybase.connect(...)  # return Connection

# in thread1
c1 = db.cursor()
c1.execute('select * blah')
while 1:
    row = c1.fetchone()
    if not row:
        break

# in thread2
c2 = db.cursor()
c2.execute('select * from blahblah')
while 1:
    row = c2.fetchone()
    if not row:
        break

Now the problem is that even though the database connection can be
shared between threads it can only support a single query in flight at
a time.  This means that while a cursor is fetching results over a
Connection I need the Cursor to maintain a lock on the Connection.  I
do not have any control over the structure of the client code which is
using the Cursor.

In my code the Cursor obtains a lock on the Connection at the start of
a result set and releases the lock at the end of the result.  This has
allowed one user to implement a multi-threaded server which
transparently shares a limited number of database connections between
a much larger set of client handling threads.

The __del__ requirement comes in because the DB-API does not require
that you complete result fetching with a cursor.  It also does not
require that you close a cursor.  This means that the following is a
perfectly fine program:

db = Sybase.connect(...)
c = db.cursor()
c.execute('select * reallbigtable')
row = c.fetchone()

Client code does not need to perform any cleanup.

- Dave

-- 
http://www.object-craft.com.au



More information about the Python-list mailing list