Threading / Queue management

Power Button mjbuchan at gmail.com
Mon Feb 2 19:55:50 CET 2009


hi there,

I wonder if anyone can help with the following. I have written a
script which polls a server and if it finds and pending orders, it
instantiates an new object (foo) - in a new thread and processes some
data. In the new object (foo), there are also some long running
processes so I am trying to create a thread pool/queue and to push
items onto the queue in the instantiated object (foo). Currently I am
creating the Queue in foo but this means I am starting up 5 new
threads every time I instantiate foo. What I want to be able to do is
create the Queue and start 5 (n) threads when my program starts and
then push items onto the queue in foo.

My question is, how can I create the Queue in my main object and set
the target function for the Thread Constructor to be a function in
foo?

Below is a code snippet if it will help explain more - as I am fairly
new to Python.

Thanks.
M


import os, threading, time
import Queue
class Foo(threading.Thread):

    def __init__(self, record):
        self.mRecord = record
        self.mQueue = None
        threading.Thread.__init__(self)

    def run(self):
	    files = self.getFiles()
	    self.sendFiles(files)
	    #snip

    def getFiles(self):
        #snip
        pass

    def sendFiles(self, files):
        self.mQueue = Queue.Queue()

        for i in range(5):
            t = threading.Thread(target=self.doSend)
            t.setDaemon(False)
            t.start()

         for f in files:
             self.mQueue.put(['ok', f])

        self.mQueue.join()

    def doSend(self):
        flag = 'ok'
        while flag != 'stop':
            flag,item = self.mQueue.get()
            if flag == 'ok':
                self.sendIt(item)

            self.mQueue.task_done()

    def sendIt(self, item):
        #snip
        pass


import sys, traceback, time
from modules.foo import Foo
from modules.bar import Bar
class Server(object):

    def startPolling(self):
        while True:
            bar = Bar(accessKey)
            pending = None
            pending = bar.getPending()

            if pending != None:
                self. processRecord(bar)
            time.sleep(10)


    def processRecord(self, record):
        Foo(record).start()



def main():
    server = Server()
    server.startPolling()

if __name__ == '__main__':
    main()



More information about the Python-list mailing list