threads or queue for this task

[James J. Besemer]

robin at execulink.com wrote:

>I may sometimes be clever, and can no doubt cobble together code from
>reading the docs (which I have been doing so far), but what I don't
>know are some of the trade-offs of using certain constructs, or the
>optimal way to do things. That is much more difficult to glean without
>actual code examples.
>
>For example, I really don't know what "proper design" I should use in
>this case. Any sample code?
>
The following illustrates my understanding of what you're trying to do.

    import thread
    import Queue

    # three queues connect four threads

    CmdQ1 = Queue.Queue()
    CmdQ2 = Queue.Queue()
    CmdQ3 = Queue.Queue()

    # Reader thread reads commands from stdin and
    # dispatches to appropriate thread via queue

    def Reader():
        while 1:
            line = stdin.readline()
            if not line:
                return

            if line[0] == '1':
                CmdQ1.put( line[1:])
            elif line[0] == '2':
                CmdQ2.put( line[1:])
            elif line[0] == '3':
                CmdQ3.put( line[1:])
            else:
                continue    # or signal error

    # Command threads process commands as they arrive via queue

    def Cmd1():
        while 1:
            cmd = CmdQ1.get()
            process1( cmd )

    def Cmd2():
        while 1:
            cmd = CmdQ2.get()
            process2( cmd )

    def Cmd3():
        while 1:
            cmd = CmdQ3.get()
            process3( cmd )

    def main():
       
        thread.start_new_thread( Cmd1, ())
        thread.start_new_thread( Cmd2, ())
        thread.start_new_thread( Cmd3, ())

        # reader thread runs in mainline
        Reader()


Frankly, it doesn't get any simpler than this.

With threads as nodes and Queues as pipes, you can create practically 
any data flow topology you wish.

>The interface to Queue is deceptively simple. Are there constructs to
>avoid or performance trade-offs to different methods? Again, any
>explicit code (or at least explicit help as to which methods are
>preferable) would be appreciated.
>

It's not deceptively simple.  It's merely simple.  Sometimes a Queue is 
simply a Queue.

>I am also put off by the description of the Queue Object, particularly
>methods like empty() which says: "Return 1 if the queue is empty, 0
>otherwise. Because of multithreading semantics, this is not reliable."
>

The answer is accurate when it is returned to you but due to the nature 
of multithreading semantics, the value also is subject to change without 
notice.  So if you read it and see a 0 or a 1, this is a true historical 
fact but you can't rely on the value being true at arbitrary points in 
the future so you can't really base any decisions on it.  

This is a function that was trivial and natural for the class designer 
to implement.  It probably IS used internal to the class in order to 
decide whether or not to suspend or resume a thread.  But is of little 
practical value to the end user.  Fageddabouit.

>How can one code if you don't
>reliably know when a Queue is empty or full?
>

Generally, Queues never get "full" and you generally don't care if 
they're empty.  

The import thing about Queues is that the reader thread will 
automatically be suspended when reading from an empty queue and 
furthermore it will automatically wake up when the queue becomes non empty.

All you need from Queues is to create them and to get and put data.

>What this boils down to is: I'm sure I can do the job with the great
>help I've already received, but do not have the confidence to know if
>I'm doing the job the *right* way.
>

If your code does what you want and doesn't take an unreasonable amount 
of time to run on your computer then that's 80% of the "*right*" way. 
 I'll defer to others to argue the remaining 20%.

There really aren't as many alternatives here as you seem to suspect.

You probably should look for a book on concurrent sequential processes. 
 Most of the paradigms and pitfalls are pretty standard and would be 
spelled out in the book.  Unfortunately, I am at a loss to make a 
specific recommendation.  Perhaps somebody else has an idea.

Regards

--jb

-- 
James J. Besemer		503-280-0838 voice
2727 NE Skidmore St.		503-280-0375 fax
Portland, Oregon 97211-6557	mailto:jb at cascade-sys.com
				http://cascade-sys.com