[Tutor] Unit testing

[Terry Carroll]

On Wed, 28 Jun 2006, Terry Carroll wrote:

> On Wed, 28 Jun 2006, Tino Dai wrote:
> 
> > Ok, I think I'm going to back up and explain what I'm am heading towards.
> > I'm working on an app that fire off a bunch of threads. Each one of these
> > threads in connected via queues to another thread in a sequence like a
> > chain. And how I tell the next stage thread that there is data in the queue
> > is via semaphore. 
> 
> You can just use a series of Queues, where each Queue represents the work 
> being passed from one thread to the other....

Okay, I was bored tonight, so I cooked up an illustration.

Here's an example with five stages.  Stage 1 takes a string and fills an 
input queue with a series of letters from the string.  Stages 2-4 do just 
take a letter off its input queue and move it to its output queue.  Stage 
5 takes a letter off its input queue and then assembles it into the string 
and prints it once complete.

A token "stop" rather than a letter on the queue is a signal to shut down.  
Each stage recognizes the token when received, passes it onto its output 
queue (if any) and then shuts down.

Status messages show the progress and threading; a pause of random 
duration keeps things from being too predictable.

Code:

import time, Queue, threading, random

def randompause():
    time.sleep(random.randint(1,5))

def statusmessage(id, workunit):
    print "%s thread %s, workunit %s" %(time.asctime(), id, workunit)

def endmessage(id):
    print "%s thread %s ending" %(time.asctime(), id)

class processor(threading.Thread):
    def __init__(self, id, inQ=None, outQ=None, inmessage=None):
        """
        id is thread Id: "first" for initial producer,
          "last" for final consumer
           inQ is input Q, or None for first producer
           outQ is outputQ, or None for final consumer
        """
        self.inQ = inQ
        self.outQ = outQ
        self.id = id
        if self.id == "first":
            self.l = list(inmessage)
        if self.id == "last":
            self.message=""
        threading.Thread.__init__(self)

    def producer(self):
        while True:
            randompause()
            try:
                workunit = self.l.pop(0)
            except IndexError:
                self.outQ.put("stop")
                endmessage(self.id)
                return
            statusmessage(self.id, workunit)
            self.outQ.put(workunit)

    def consumer(self):
        while True:
            randompause()
            workunit = self.inQ.get(True)
            if workunit == "stop":
                print "final message:", self.message
                endmessage(self.id)
                return
            else:
                statusmessage(self.id, workunit)
                self.message = self.message+workunit

    def hybrid(self):
        while True:
            randompause()
            workunit = self.inQ.get(True)
            if workunit == "stop":
                self.outQ.put(workunit)
                endmessage(self.id)
                return
            else:
                statusmessage(self.id, workunit)
                self.outQ.put(workunit)

    def run(self):
        if self.id == "first":
            processor.producer(self)
        elif self.id == "last":
            processor.consumer(self)
        else:
            processor.hybrid(self)

if __name__ == "__main__":
    q_ab = Queue.Queue()
    pa = processor(id="first", inmessage="spam", outQ=q_ab)
    q_bc = Queue.Queue()
    pb = processor(id="second", inQ=q_ab, outQ=q_bc)
    q_cd = Queue.Queue()
    pc = processor(id="third", inQ=q_bc, outQ=q_cd)
    q_de = Queue.Queue()
    pd = processor(id="fourth", inQ=q_cd, outQ=q_de)
    pe = processor(id="last", inQ=q_de)
    pa.start()
    pb.start()
    pc.start()
    pd.start()
    pe.start()
    
Result:
    
Thu Jun 29 00:37:30 2006 thread first, workunit s
Thu Jun 29 00:37:31 2006 thread first, workunit p
Thu Jun 29 00:37:32 2006 thread first, workunit a
Thu Jun 29 00:37:33 2006 thread second, workunit s
Thu Jun 29 00:37:33 2006 thread third, workunit s
Thu Jun 29 00:37:33 2006 thread fourth, workunit s
Thu Jun 29 00:37:33 2006 thread last, workunit s
Thu Jun 29 00:37:34 2006 thread first, workunit m
Thu Jun 29 00:37:37 2006 thread second, workunit p
Thu Jun 29 00:37:37 2006 thread third, workunit p
Thu Jun 29 00:37:38 2006 thread first ending
Thu Jun 29 00:37:38 2006 thread fourth, workunit p
Thu Jun 29 00:37:38 2006 thread last, workunit p
Thu Jun 29 00:37:41 2006 thread second, workunit a
Thu Jun 29 00:37:41 2006 thread third, workunit a
Thu Jun 29 00:37:41 2006 thread fourth, workunit a
Thu Jun 29 00:37:41 2006 thread last, workunit a
Thu Jun 29 00:37:43 2006 thread second, workunit m
Thu Jun 29 00:37:43 2006 thread third, workunit m
Thu Jun 29 00:37:43 2006 thread fourth, workunit m
Thu Jun 29 00:37:44 2006 thread last, workunit m
Thu Jun 29 00:37:47 2006 thread second ending
Thu Jun 29 00:37:47 2006 thread third ending
Thu Jun 29 00:37:48 2006 thread fourth ending
final message: spam
Thu Jun 29 00:37:48 2006 thread last ending