[Tutor] glibc error while Python script runs - Solved

[Bernard Lebel]

Thanks a lot Danny,

That certainly does make sense. I'll look into implementing the Queue
approach in my program tomorrow. I remember you recommending me this
module as well not long ago, although in a different discussion (where
I suspected problem with file access from multiple thread, but I guess
it's pretty much a similar problem, isn't it? :-)


Thanks again
Bernard



On 1/19/06, Danny Yoo <dyoo at hkn.eecs.berkeley.edu> wrote:
>
>
> On Thu, 19 Jan 2006, Kent Johnson wrote:
>
>
> > In your original desing were you sharing a connection between threads?
> > That could cause trouble. But if each connection has its own thread and
> > you are using transactions and isolation levels appropriately, they
> > shouldn't stomp on each other.
>
> Hi Kent and Bernard,
>
> It's possible that MySQLdb was linked against the non-thread-safe
> mysqlclient (rather than mysqlclient_r) library; that would probably cause
> havoc.
>
>
> > > So the solution was to start some sort of queue server in a separate
> > > thread. This queue would consist of a list, and each time the program
> > > would want to perform a MySQL operation, it would add it to the queue.
>
> We may want to use the Queue module here instead of a list; the way the
> program is described now sounds like the server is busy-spinning when it
> looks for new things to do.  The thing that makes busy-spinning slightly
> not-nice is that it consumes CPU regardless if the system's doing anything
> or not.
>
> But a more serious problem than busy-waiting is one of thread-safety: the
> object that's used to communicate between two threads --- a shared list
> --- might be unreliable if the list methods aren't thread-safe.  And I'm
> not certain that all the list methods are thread-safe.
>
> That is, the problem with mutual exclusion may have just been pushed up
> the program's structure, from the MySQL queries up to the shared queue
> list access.  *grin*
>
>
> The synchronized Queue described in:
>
>     http://www.python.org/doc/lib/module-Queue.html
>
> is designed to be a reliable communication medium between threads, and I
> strongly recommend you look at it, especially because you've seen
> first-hand the weird things that can happen with threads.  *grin*
>
>
> Here's a small example that shows what a different Queue can make:
>
> ############################################
> from threading import Thread
>
> class Server:
>     def __init__(self):
>         self.queue = []
>
>     def acceptJob(self, query):
>         self.queue.append((query,))
>
>     def shutdownOnIdle(self):
>         self.queue.append("QUIT!")
>
>     def jobLoop(self):
>         while True:
>             print "looping"
>             if len(self.queue) > 0:
>                nextJob = self.queue.pop(0)
>                if nextJob == "QUIT!":
>                    return
>                print "I should do", nextJob
>
>     def startServer(self):
>         Thread(target=self.jobLoop).start()
>
> if __name__ == '__main__':
>     server = Server()
>     server.startServer()
>     server.acceptJob("a")
>     server.acceptJob("b")
>     server.acceptJob("c")
>     server.shutdownOnIdle()
> ############################################
>
>
> Running this will show just how much work a busy-waiting thread does.
>
>
> But if we change a few methods to use Queues instead of lists:
>
> ######
> from threading import Thread
> from Queue import Queue
>
> class Server:
>     def __init__(self):
>         self.queue = Queue()
>
>     def acceptJob(self, query):
>         self.queue.put((query,))
>
>     def shutdownOnIdle(self):
>         self.queue.put("QUIT!")
>
>     def jobLoop(self):
>         while True:
>             print "looping"
>             nextJob = self.queue.get()
>             if nextJob == "QUIT!":
>                 return
>             print "I should do", nextJob
>
>     def startServer(self):
>         Thread(target=self.jobLoop).start()
>
> if __name__ == '__main__':
>     server = Server()
>     server.startServer()
>     server.acceptJob("a")
>     server.acceptJob("b")
>     server.acceptJob("c")
>     server.shutdownOnIdle()
> ######
>
> and compare the output of this to the original implementation, it should
> be clearer why Queues are cool.  *grin*
>
>
> Does this make sense?  I hope this helps!
>
>