Beginner Threaded file reading access

Jeff Epler jepler at unpythonic.net
Tue Mar 23 19:17:57 CET 2004


First, make sure that you're adding threads in a way that will give you
better performance.  The program below speeds up dramatically, taking
about 12 seconds for nworkers=5 and 2 seconds for nworkers=50.  Of
course, just doing nothing takes about 0 seconds if you write it
correctly.

Anyway, you want to create a Queue.Queue which is available to your
N consumer threads and your producer thread (in my example, this is the
main thread).  You also want to have a special value that causes the
consumer threads to exit, when the work is complete.

Trying to iterate over the same file from multiple threads will make you
cry.  Iterate over the file in the producer, not the consumer.

Jeff

import Queue, threading, time, random
nworkers = 5
Shutdown = object()
queue = Queue.Queue(2*nworkers)

def worker(i, queue, Shutdown):
    while 1:
        work = queue.get()
        if work is Shutdown: break
        print "Working on", work, "from thread", i
        time.sleep(random.random())

def producer():
    for i in range(100): queue.put(i)

def create_threads():
    return [threading.Thread(target=worker, args=(i, queue, Shutdown))
                        for i in range(nworkers)]

def end_threads(workers):
    for t in workers:
        queue.put(Shutdown)
    for t in workers:
        t.join()

threads = create_threads()
for t in threads: t.start()
producer()
end_threads(threads)





More information about the Python-list mailing list