[Tutor] glibc error while Python script runs - Solved

[Danny Yoo]

On Thu, 19 Jan 2006, Kent Johnson wrote:


> In your original desing were you sharing a connection between threads?
> That could cause trouble. But if each connection has its own thread and
> you are using transactions and isolation levels appropriately, they
> shouldn't stomp on each other.

Hi Kent and Bernard,

It's possible that MySQLdb was linked against the non-thread-safe
mysqlclient (rather than mysqlclient_r) library; that would probably cause
havoc.


> > So the solution was to start some sort of queue server in a separate
> > thread. This queue would consist of a list, and each time the program
> > would want to perform a MySQL operation, it would add it to the queue.

We may want to use the Queue module here instead of a list; the way the
program is described now sounds like the server is busy-spinning when it
looks for new things to do.  The thing that makes busy-spinning slightly
not-nice is that it consumes CPU regardless if the system's doing anything
or not.

But a more serious problem than busy-waiting is one of thread-safety: the
object that's used to communicate between two threads --- a shared list
--- might be unreliable if the list methods aren't thread-safe.  And I'm
not certain that all the list methods are thread-safe.

That is, the problem with mutual exclusion may have just been pushed up
the program's structure, from the MySQL queries up to the shared queue
list access.  *grin*


The synchronized Queue described in:

    http://www.python.org/doc/lib/module-Queue.html

is designed to be a reliable communication medium between threads, and I
strongly recommend you look at it, especially because you've seen
first-hand the weird things that can happen with threads.  *grin*


Here's a small example that shows what a different Queue can make:

############################################
from threading import Thread

class Server:
    def __init__(self):
        self.queue = []

    def acceptJob(self, query):
        self.queue.append((query,))

    def shutdownOnIdle(self):
        self.queue.append("QUIT!")

    def jobLoop(self):
        while True:
            print "looping"
            if len(self.queue) > 0:
               nextJob = self.queue.pop(0)
               if nextJob == "QUIT!":
                   return
               print "I should do", nextJob

    def startServer(self):
        Thread(target=self.jobLoop).start()

if __name__ == '__main__':
    server = Server()
    server.startServer()
    server.acceptJob("a")
    server.acceptJob("b")
    server.acceptJob("c")
    server.shutdownOnIdle()
############################################


Running this will show just how much work a busy-waiting thread does.


But if we change a few methods to use Queues instead of lists:

######
from threading import Thread
from Queue import Queue

class Server:
    def __init__(self):
        self.queue = Queue()

    def acceptJob(self, query):
        self.queue.put((query,))

    def shutdownOnIdle(self):
        self.queue.put("QUIT!")

    def jobLoop(self):
        while True:
            print "looping"
            nextJob = self.queue.get()
            if nextJob == "QUIT!":
                return
            print "I should do", nextJob

    def startServer(self):
        Thread(target=self.jobLoop).start()

if __name__ == '__main__':
    server = Server()
    server.startServer()
    server.acceptJob("a")
    server.acceptJob("b")
    server.acceptJob("c")
    server.shutdownOnIdle()
######

and compare the output of this to the original implementation, it should
be clearer why Queues are cool.  *grin*


Does this make sense?  I hope this helps!