Threading problems at program exit

[Ype Kingma]

Dave Cole wrote:

> 
> Dave> Is there a better or more kosher way?
> 
> Ype> You might consider not using the __del__ method at all because
> Ype> 'it is not guaranteed that __del__() methods are called for
> Ype> objects that still exist when the interpreter exits. ' (quoted
> Ype> from the language ref).  Then use:
> Ype>
> Ype> obj = Locked(threading.RLock())
> Ype> try:
> Ype>      # whatever needs to be done with obj
> Ype> finally:
> Ype>     obj.unlock()
> 
> That won't work in this case because the resource needs to be locked
> over a span of client code.
> 
> I should be more specific...
> 
> The code in question is my Python bindings for Sybase.  According the
> the DB-API specification a database module must provide Connection
> objects for managing database connections, and Cursor objects for
> executing commands over a Connection.  The DB-API implies that you
> should be able to do something like the following:
> 
> db = Sybase.connect(...)  # return Connection
> 
> # in thread1
> c1 = db.cursor()
> c1.execute('select * blah')
> while 1:
>     row = c1.fetchone()
>     if not row:
>         break
> 
> # in thread2
> c2 = db.cursor()
> c2.execute('select * from blahblah')
> while 1:
>     row = c2.fetchone()
>     if not row:
>         break

Is there an implicit requirement that the cursor is used on a single
thread?
Since these threads do not have to signal that they are finished using
their cursor, the only option you have is to wait for  one connection using
thread to exit when another connection using thread wants to 
use the connection.
You could do this by join()'ing the first thread from the other thread.
 
> Now the problem is that even though the database connection can be
> shared between threads it can only support a single query in flight at
> a time.  This means that while a cursor is fetching results over a
> Connection I need the Cursor to maintain a lock on the Connection.  I
> do not have any control over the structure of the client code which is
> using the Cursor.
> 
> In my code the Cursor obtains a lock on the Connection at the start of
> a result set and releases the lock at the end of the result.  This has
> allowed one user to implement a multi-threaded server which
> transparently shares a limited number of database connections between
> a much larger set of client handling threads.

You could also allow another thread to use connection at the 'end of 
result'. This is much nicer than waiting for threads to exit, but
if a thread chooses not to process until 'end of result' you will get
into this situation.

> The __del__ requirement comes in because the DB-API does not require
> that you complete result fetching with a cursor.  It also does not
> require that you close a cursor.  This means that the following is a
> perfectly fine program:

The problem is that __del__ is _not_ guaranteed to be called, so you can't
use it for hard requirements. (I mostly use Jython, and there __del__
is at the mercy of the Java garbage collector. It does have it's uses,
but mostly for coping with memory pressure.)

There is one more option you have. You might require that, when another 
thread is waiting for the connection, a thread either exits or does some 
more processing on the cursor within a reasonable
amount of time Failing that you might allow other threads to use the 
connection and throw a  time out exception the next time the slow thread 
uses the cursor.

As this might happen in a hard to debug situation, you might want to
log the time out and the slow thread.
At the server side you might have to roll back the transaction of the slow 
thread.

Have fun :),
Ype