multiprocessing timing issue

[Tim Arnold]

On 8/10/2011 11:36 PM, Philip Semanchuk wrote:
>
> On Aug 9, 2011, at 1:07 PM, Tim Arnold wrote:
>
>> Hi, I'm having problems with an empty Queue using multiprocessing.
>>
>> The task:
>> I have a bunch of chapters that I want to gather data on individually and then update a report database with the results.
>> I'm using multiprocessing to do the data-gathering simultaneously.
>>
>> Each chapter report gets put on a Queue in their separate processes. Then each report gets picked off the queue and the report database is updated with the results.
>>
>> My problem is that sometimes the Queue is empty and I guess it's
>> because the get_data() method takes a lot of time.
>>
>> I've used multiprocessing before, but never with a Queue like this.
>> Any notes or suggestions are very welcome.
>
>
> Hi Tim,
> THis might be a dumb question, but...why is it a problem if the queue is empty? It sounds like you figured out already that get_data() sometimes takes longer than your timeout. So either increase your timeout or learn to live with the fact that the queue is sometimes empty. I don't mean to be rude, I just don't understand the problem.
>
> Cheers
> Philip
>

Hi Philip,
Not a dumb or rude question at all, thanks for thinking about it. When 
the queue is empty the report cannot be updated, so that's why I was 
concerned--I couldn't figure out how to block. Now that's dumb!

 From your response and Tim Roberts too, I see that it's possible to 
block until the data comes back. I just should never have put that 
timeout in there. I must have assumed it would not block with no timeout 
given. Wrong....

 From the docs on q.get():
If optional args 'block' is true and 'timeout' is None (the default),
block if necessary until an item is available. If 'timeout' is
a positive number, it blocks at most 'timeout' seconds and raises
the Empty exception if no item was available within that time.
Otherwise ('block' is false), return an item if one is immediately
available, else raise the Empty exception ('timeout' is ignored
in that case).

thanks,
--Tim Arnold