threading question

[MRAB]

Mag Gam wrote:
> Hello,
> 
> Currently, I am using a bash script to ssh into 400 servers and get an
> output, ie check if a file exists on a local filesystem.  I am doing
> this linearly, however I am interesting in doing this with threads and
> more important using Python standard threading library.
> 
> My pseudo code would be something like this:
> 
> Have a list of the servers in a dictionary
> Create the number of threads which correspond to servers.  ie 400 threads
> Each thread will spawn ssh and execute the file test
> Once all threads are completed, I send a signal saying "all is one",
> and it will create some sort of output.
> 
> 
> Is there a better way of doing this? My goal is to learn python and
> threading, I know this is a silly example but its worth a try :-)
> 
I wouldn't use that many threads. It's better to use a relatively small
number of them (not more than 10, perhaps?) and let each thread do
multiple jobs, one at a time.

Here some example code:


# For Python 2.5 and later

from threading import Thread
from Queue import Queue

class ThreadTask(object):
     def __init__(self, task_queue):
         self.task_queue = task_queue
     def __call__(self):
         while True:
             task = self.task_queue.get()
             if task is None:
                 # A None indicates that there are no more tasks.
                 break
             self.process_task(task)
         # Put back the None so that the next thread will also see it.
         self.task_queue.put(None)
     def process_task(self, task):
         # Put your code here
         ...

MAX_RUNNING = 10

# Put the tasks into a queue, ending with a None.
task_queue = Queue()
for t in task_list:
     task_queue.put(t)
task_queue.put(None)

# Start the threads to do the work.
thread_list = []
for i in range(MAX_RUNNING):
     t = Thread(target=ThreadTask(task_queue))
     t.start()
     thread_list.append(t)

# The threads will finish when they see the None in the task queue.
for t in thread_list:
     t.join()

# When you get here all the task will have finished.