[ python-Bugs-1080660 ] thread.error: release unlocked lock on Queue put

[SourceForge.net]

Bugs item #1080660, was opened at 2004-12-07 14:48
Message generated for change (Comment added) made by snacktime
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1080660&group_id=5470

Category: Threads
Group: Python 2.3
Status: Closed
Resolution: Later
Priority: 5
Submitted By: John Speno (corvus)
Assigned to: Nobody/Anonymous (nobody)
Summary: thread.error: release unlocked lock on Queue put

Initial Comment:
System is Python 2.3.3 on Solaris 9. 

I'm using the classic Python thread model. Main thread puts stuff into
a Queue.Queue() and worker threads get stuff out of it and do their 
thing.

On occaision, I get an exception in the main thread when it tries to
put something into the Queue.

   File "/usr/local/bin/poller.py", line 47, in fragnap_it
     work_queue.put((foo, bar, baz))
   File "/usr/local/lib/python2.3/Queue.py", line 106, in put
     self.fsema.release()
   thread.error: release unlocked lock

This error still happens intermittently, and I haven't been able to 
reduce it to a simple case. I can alter my applications to do 
something useful for debugging this problem when it happens, if you 
can figure out what "something useful" would be. Just let me know.

The queue is unbounded, and I'm using blocking get/put.

The main thread creates 20 worker threads, and 1 recorder thread. 
Then it puts upto 1500 items in the Queue.Queue named 
work_queue. The 20 worker threads take stuff out work_queue, 
process some using pure python code, or others by using 
subprocess.py. Results from the workers are put into another 
Queue.Queue named results_queue. The Recorder thread gets from 
the results_queue and writes stuff to a MySQL db. I've never seen the 
error happen when putting into the results_queue, only from the main 
thread putting into the work_queue.

Thread defines in my python config.h are:

#define HAVE_PTHREAD_H 1
#define HAVE_PTHREAD_SIGMASK 1
#define HAVE_THREAD_H 1
#define PTHREAD_SYSTEM_SCHED_SUPPORTED 1
#define SIZEOF_PTHREAD_T 4
#define WITH_THREAD 1

And all the thread related python tests (test_thread, test_threading, 
and test_queue) pass.

----------------------------------------------------------------------

Comment By: Chris Ochs (snacktime)
Date: 2005-02-08 04:51

Message:
Logged In: YES 
user_id=31170

Me being that someone....  Here is some extra information in 
addition to what is in the referenced post on twisted-python.

Besides twisted, the only other third party modules I am using 
are psycopg and pycrypto.  I'm going to pull out the code 
that uses those modules one at a time and see if I can 
narrow it down.  The segfaults/bus errors are easy to repeat 
and happen every time unless gdb is attached, in which case 
I get the exception.  For some reason it seems like gdb does 
something that makes the code return the exception instead 
of segfault.

I'll try to put together the smallest test case I can that 
segfaults and post it here.  

Chris
chris at paymentonline.com

----------------------------------------------------------------------

Comment By: Andrew Bennetts (spiv)
Date: 2005-02-08 04:33

Message:
Logged In: YES 
user_id=50945

Looks like someone is seeing something similar on FreeBSD
and Python 2.4, with the added bonus of segfaults:
   
http://twistedmatrix.com/pipermail/twisted-python/2005-February/009474.html

They are apparently able to reproduce the problem, too --
but a 3rd-party extension module (psycopg) is involved, so
who knows...


----------------------------------------------------------------------

Comment By: John Speno (corvus)
Date: 2004-12-10 17:57

Message:
Logged In: YES 
user_id=2138

Given that I can't reproduce it in a simple test, and it happens so rarely in 
my real applications, I'm okay with notice of it just being in google and here 
on sf. Maybe someday someone will encounter it again and we can make 
progress.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-12-10 17:45

Message:
Logged In: YES 
user_id=31435

No, and there are no known bugs in Queue in 2.3.j.  If John 
can't whittle it down to a test case that fails for others, I 
expect this bug is hopeless.  We talked about it on c.l.py, 
and what he's seeing should be impossible.  Presuming it's a 
Solaris-specific bug.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-12-10 17:35

Message:
Logged In: YES 
user_id=80475

Tim, was this something you already fixed in Py2.4?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1080660&group_id=5470