Beginner Threaded file reading access

[Jeff Epler]

First, make sure that you're adding threads in a way that will give you
better performance.  The program below speeds up dramatically, taking
about 12 seconds for nworkers=5 and 2 seconds for nworkers=50.  Of
course, just doing nothing takes about 0 seconds if you write it
correctly.

Anyway, you want to create a Queue.Queue which is available to your
N consumer threads and your producer thread (in my example, this is the
main thread).  You also want to have a special value that causes the
consumer threads to exit, when the work is complete.

Trying to iterate over the same file from multiple threads will make you
cry.  Iterate over the file in the producer, not the consumer.

Jeff

import Queue, threading, time, random
nworkers = 5
Shutdown = object()
queue = Queue.Queue(2*nworkers)

def worker(i, queue, Shutdown):
    while 1:
        work = queue.get()
        if work is Shutdown: break
        print "Working on", work, "from thread", i
        time.sleep(random.random())

def producer():
    for i in range(100): queue.put(i)

def create_threads():
    return [threading.Thread(target=worker, args=(i, queue, Shutdown))
                        for i in range(nworkers)]

def end_threads(workers):
    for t in workers:
        queue.put(Shutdown)
    for t in workers:
        t.join()

threads = create_threads()
for t in threads: t.start()
producer()
end_threads(threads)