using queue

[MRAB]

Tim Arnold wrote:
> "Jan Kaliszewski" <zuo at chopin.edu.pl> wrote in message 
> news:mailman.895.1251958800.2854.python-list at python.org...
>> 06:49:13 Scott David Daniels <Scott.Daniels at acm.org> wrote:
>>
>>> Tim Arnold wrote:
>>>> (1) what's wrong with having each chapter in a separate thread? Too 
>>>> much going on for a single processor?
>>> Many more threads than cores and you spend a lot of your CPU switching
>>> tasks.
>> In fact, python threads work relatively the best with a powerful single
>> core; with more cores it becomes being suprisingly inefficient.
>>
>> The culprit is Pythn GIL and the way it [mis]cooperates with OS
>> scheduling.
>>
>> See: http://www.dabeaz.com/python/GIL.pdf
>>
>> Yo
>> *j
>>
>> -- 
>> Jan Kaliszewski (zuo) <zuo at chopin.edu.pl>
> 
> I've read about the GIL (I think I understand the problem there)--thanks. In 
> my example, the actual job called for each chapter ended up being a call to 
> subprocess (that called a different python program). I figured that would 
> save me from the GIL problems since each subprocess would have its own GIL.
> 
> In the words of Tom Waits, " the world just keeps getting bigger when you 
> get out on your own".  So I'm re-reading now, and maybe what I've been doing 
> would have been better served by the multiprocessing package.
> 
> I'm running python2.6 on FreeBSD with a dual quadcore cpu. Now my questions 
> are:
> (1) what the heck should I be doing to get concurrent builds of the 
> chapters, wait for them all to finish, and pick up processing the main job 
> again? The separate chapter builds have no need for communication--they're 
> autonomous.
> (2) using threads with the target fn calling subprocess, a bad idea?
> (3) should I study up on multiprocessing package and/or pprocessing?
> 
> thanks for your inputs,
> 
You could adapt the threading solution I gave to multiprocessing; just
use the multiprocessing queue class instead of the threading queue
class, etc.