Limit number of concurrent threads

[Chuck May]

Ok...I think I've got it.  Thanks for everyone's help.

How does this look:

from threading import Thread
import Queue, time, random

maxthreads =  5  # maximum number of concurrent threads
totalops   = 80  # hold total number of operations

class Worker(Thread):
    def __init__(self, q):
        Thread.__init__(self)
        self.q  = q

    def run(self):
        all_done = 0
        while not all_done:
            try:
                table = self.q.get(0)
                time.sleep(random.randint(10, 1000) / 1000.0)
                print "id %d" % table
            except Queue.Empty:
                all_done = 1

if __name__ == "__main__":
    q = Queue.Queue()
    for i in range(totalops):
        q.put(i)

    threads = []
    for i in range(maxthreads):
        t = Worker(q)
        threads.append(t)
        t.start()

    # wait for all threads to complete
    for t in threads:
        t.join();

    print "Al done!"

Syver Enstad <syver-en+usenet at online.no> wrote in
news:uwupbdwnj.fsf at online.no: 

> Chuck May <cmay4 at yahoo.com> writes:
> 
>> I've searching for examples that fit my problem, and I haven't found 
>> any.  Most examples I've found look something like this:
>> 
>>     threads = []
>>     for i in range(number_of_tasks):
>>         thread.append(thread_for_task)
>>     
>>     # let the threads run - just wait for shutdown
>>     for t in threads:
>>         t.join() 
>> 
>> which whould work fine if I wanted to start all 80 threads at once.
>> 
>> I've read about the Queue, but I'm not sure if it's what I want, and
>> I 
>> 
>> can't seem to find any good examples on it.
> 
> I think Queue would be nice to use, Have a thread (the main thread
> perhaps) feed all the table names (or a table object if you please)
> into a Queue. 
> 
> Set up some worker threads with access to the Queue
> (global variable, member). The worker threads should loop something
> like this:
> 
> while 1:
>    queueItem = self._queue.get()
>    doSomethingWithQueueItemHere
> 
> Then you can start your workers.
> 
> (You'll have to find out some way to make the workers exit when the
> work is done, perhaps putting a number of None's equal to the number
> of workers last in the queue and testing for None in the working
> thread loop. 
> 
> 



-- 
Chuck May
IMS, Inc.