[Python-bugs-list] [ python-Bugs-544473 ] Queue module can deadlock

noreply@sourceforge.net noreply@sourceforge.net
Thu, 18 Apr 2002 17:12:51 -0700


Bugs item #544473, was opened at 2002-04-16 12:09
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=544473&group_id=5470

Category: Python Library
Group: None
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Mark Hammond (mhammond)
Assigned to: Mark Hammond (mhammond)
Summary: Queue module can deadlock

Initial Comment:
Prompted by a users experimentation where he was able
to show Pythonwin hanging, I have found a case where
the Queue module deadlocks.  It appears that if a
RuntimeError is generated while queueing data, the
queue lock will remain held.

Attaching a test script.  This hangs on Win2k and
Linux.  Win2k callstack shows Queue is waiting for the
following lock (line 53)

        if block:
            self.fsema.acquire()

Presumably (but untested at the moment), the previous
call to q.put(), which triggered a "RuntimeError:
maximum recursion depth exceeded", left the lock acquired.

It seems a few choice try/finally statements in the
Queue module would go along way :)  I will try and get
to this soon.



----------------------------------------------------------------------

>Comment By: Mark Hammond (mhammond)
Date: 2002-04-19 10:12

Message:
Logged In: YES 
user_id=14198

Checking in Queue.py;
/cvsroot/python/python/dist/src/Lib/Queue.py,v  <--  Queue.py
new revision: 1.15; previous revision: 1.14
Checking in test/test_queue.py;
/cvsroot/python/python/dist/src/Lib/test/test_queue.py,v 
<--  test_queue.py
initial revision: 1.1

bugfix candidate.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-19 04:20

Message:
Logged In: YES 
user_id=31435

Changed to Accepted to encourage you to check it in <wink>.

Note that expected-output files are not *required* for 
tests anymore, and indeed are discouraged.  Better to print 
purely informational msgs under a

if test_support.verbose:
    print "yadda, yadda"

guard, and not create any expected-output file.

In the checkin msg(s), be sure to say this is a "bugfix 
candidate" patch.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2002-04-18 21:30

Message:
Logged In: YES 
user_id=14198

Good idea re subclassing to provoke the deadlock!  Attaching
the exact same Queue.py patch, plus a reasonable test case -
both blocking and non-blocking gets and puts are tested in
the succeeding and failing cases.  Without the Queue.py
patch, both Linux and Windows deadlock in the tests.  With
the patch, both succeed.  If no objections I will check it in.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-18 15:09

Message:
Logged In: YES 
user_id=31435

The patch looks good to me, but I would like to see a test 
case.  Note that you can build a reliable test case by 
subclassing Queue and overriding its _put and/or _get 
methods, arranging to raise an exception deliberately from 
them.  Any exception in these would leave a semaphore and 
self.mutex held forever in the current Queue.py.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2002-04-16 22:18

Message:
Logged In: YES 
user_id=14198

This patch seems reasonable to me.  Tim, a quick look would 
be appreciated.  Writing a reasonable test case seems 
difficult :(  It doesn't seem worth adding this little test 
script to the test suite.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-16 12:47

Message:
Logged In: YES 
user_id=31435

Yup, it hangs on Win98 too.  Nice job of whittling it down, 
Mark!  We've been looking for an excuse to make Queue 
slower too <wink>.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=544473&group_id=5470