[Tutor] Threads

[Kent Johnson]

Terry Carroll wrote:
> I'm toying with using threads for an app I have in mind.  The Queue module 
> you point out looks like just the ticket for what I'm thinking of.
> 
> One question I do have on threads, though.  In my app, a finite number 
> (12-50) requests including URLs would be put on the queue, and multiple 
> (2-4) consumer threads would pick things off the queue to process.
> 
> But what's the mechanism to tell those consumer threads that there will be 
> nothing more to consume?  It's not just an empty queue, because that could 
> happen whenever the consumers outpace the producer.  
> 
> I was toying with a special-purpose element to add to the queue, as a 
> sentinel to say "that's it, show's over; you can shut down now!"  But that 
> would take care only of the thread that happens to pull that element off 
> the queue; the other threads would just see an empty queue and have no 
> basis for knowing that this is the big final empty.  
> 
> I suppose I could ass that "shutdown element" multiple times, once for 
> each consumer thread spawned, but that seems so inelegant.
Simple and effective, though, as long as you know how many consumers 
there are.

> 
> Another technique would be to pass, in addition to the queue, a Shutdown
> flag, so that when the queue is empty and the shutdown flag is set, the
> thread shouts down.  But the problem with this is that Queue.empty()  is
> documented as not reliable.  I suppose I could do a get with a timeout,
> and check the Shutdown flag; but I can imagine there are synch issues 
> there.
This a the typical approach in Java - the worker thread checks a flag 
and returns when the flag is set. You could use a threading.Event() as 
the flag.

What happens if you just leave the consumer threads blocked on the empty 
queue? If you will need them again, or if the entire process is 
short-lived, this is a simple solution. Just mark the consumers as 
daemon threads so the app will exit while they are still alive.

You might be interested in these recipes:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65448
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/196618

Kent