multiprocessing timing issue
philip at semanchuk.com
Thu Aug 11 05:36:52 CEST 2011
On Aug 9, 2011, at 1:07 PM, Tim Arnold wrote:
> Hi, I'm having problems with an empty Queue using multiprocessing.
> The task:
> I have a bunch of chapters that I want to gather data on individually and then update a report database with the results.
> I'm using multiprocessing to do the data-gathering simultaneously.
> Each chapter report gets put on a Queue in their separate processes. Then each report gets picked off the queue and the report database is updated with the results.
> My problem is that sometimes the Queue is empty and I guess it's
> because the get_data() method takes a lot of time.
> I've used multiprocessing before, but never with a Queue like this.
> Any notes or suggestions are very welcome.
THis might be a dumb question, but...why is it a problem if the queue is empty? It sounds like you figured out already that get_data() sometimes takes longer than your timeout. So either increase your timeout or learn to live with the fact that the queue is sometimes empty. I don't mean to be rude, I just don't understand the problem.
More information about the Python-list