Limit number of concurrent threads

Martin v. Loewis martin at
Tue Sep 24 14:52:47 EDT 2002

Chuck May <cmay4 at> writes:

> I've searching for examples that fit my problem, and I haven't found 
> any.  Most examples I've found look something like this:
>     threads = []
>     for i in range(number_of_tasks):
>         thread.append(thread_for_task)
>     # let the threads run - just wait for shutdown
>     for t in threads:
>         t.join() 
> which whould work fine if I wanted to start all 80 threads at once.
> I've read about the Queue, but I'm not sure if it's what I want, and I 
> can't seem to find any good examples on it.

I would recommend to use a thread pool: create a fixed number of
threads, and have them chose tasks from a task list.

# add all jobs to the queue
q = Queue.Queue()
for j in jobs:

def process_jobs():
 while 1:
    j = q.get(block = 0)
  except Queue.Empty:

# create all worker threads
threads = []
for i in range(number_of_worker_threads):
   threading.Thread(target = process_jobs)

# join all completed threads
for t in threads:t.join()


More information about the Python-list mailing list