using queue

[Scott David Daniels]

Tim Arnold wrote:
> "MRAB" <python at mrabarnett.plus.com> wrote in message 
> news:mailman.835.1251886213.2854.python-list at python.org...
>> I don't need that many threads; just create a few to do the work and let
>> each do multiple chapters, something like this:
>>
a very pretty implementation with worker code:
>>         while True:
>>             chapter = self.chapter_queue.get()
>>             if chapter is None:
>>                 # A None indicates that there are no more chapters.
>>                 break
>>             chapter.compile()
>>         # Put back the None so that the next thread will also see it.
>>         self.chapter_queue.put(None)

and loading like:
>> for c in self.document.chapter_objects:
>>     chapter_queue.put(<some work>)
>> chapter_queue.put(None)
>> ...
>> # The threads will finish when they see the None in the queue.
>> for t in thread_list:
>>     t.join()
> 
> hi, thanks for that code. It took me a bit to understand what's going on, 
> but I think I see it now.
> Still, I have two questions about it:
> (1) what's wrong with having each chapter in a separate thread? Too much 
> going on for a single processor? 
Many more threads than cores and you spend a lot of your CPU switching
tasks.

> (2) The None at the end of the queue...I thought t.join() would just work. 
> Why do we need None?

Because your workers aren't finished, they are running trying to get
something more to do out of the queue.  The t.join() would cause a
deadlock w/o the None.

--Scott David Daniels
Scott.Daniels at Acm.Org