monitor reading file with thread

wiso gtu2003 at alice.it
Fri Jan 8 12:53:45 EST 2010


I'm reading and processing a huge file, so during the execution I want to 
now the state of the processing: how many lines are already processed, and 
so on. The first approach is:

f = open(filename)
n = 0
for l in f:
  if n % 1000 = 0:
    print "Reading %d lines" %n
  do_something(l)

but I want something diffent. I want to monitor the running of the computing 
every n seconds. It's the first time I use threading, and I think mine is 
not the best solution:

import threading
import time
import Queue

class Reader():
    def __init__(self,filename):
        self.filename = filename
        self.lineno = 0
    def __iter__(self):
        f = open(self.filename)
        for line in f:
            self.lineno += 1
            time.sleep(0.01) # slow down
            yield line
        f.close()

class Monitor(threading.Thread):
    def __init__(self,reader,stop_queue,interval=2):
        threading.Thread.__init__(self)
        self.interval = interval
        self.reader = reader
        self.stop_queue = stop_queue
    def run(self):
        while True:
            try:
                if self.stop_queue.get(timeout=self.interval) == "stop":
                    break
            except Queue.Empty:
                pass
            print "MONITOR: ", reader.lineno


reader = Reader("r1_200910.log")
q = Queue.Queue()
monitor = Monitor(reader,q)

monitor.start()
for line in reader:
    pass
    # do_somethinghard(line)
q.put("stop")
monitor.join()


It't works, but I don't like how I'm stopping the thread. In general the 
Monitor class can be more complex, for example a progress bar. Using python 
2.6



More information about the Python-list mailing list