Limit number of concurrent threads

[Martin v. Loewis]

Chuck May <cmay4 at yahoo.com> writes:

> I've searching for examples that fit my problem, and I haven't found 
> any.  Most examples I've found look something like this:
> 
>     threads = []
>     for i in range(number_of_tasks):
>         thread.append(thread_for_task)
>     
>     # let the threads run - just wait for shutdown
>     for t in threads:
>         t.join() 
> 
> which whould work fine if I wanted to start all 80 threads at once.
> 
> I've read about the Queue, but I'm not sure if it's what I want, and I 
> can't seem to find any good examples on it.

I would recommend to use a thread pool: create a fixed number of
threads, and have them chose tasks from a task list.

# add all jobs to the queue
q = Queue.Queue()
for j in jobs:
  q.put(j)

def process_jobs():
 while 1:
  try:
    j = q.get(block = 0)
  except Queue.Empty:
    return
  run_job(j)

# create all worker threads
threads = []
for i in range(number_of_worker_threads):
   threading.Thread(target = process_jobs)

# join all completed threads
for t in threads:t.join()

HTH,
Martin