multiprocessing in a while loop?

[Johan Llewellyn]

hi, I am struggling to understand how to leverage python's multiprocessing module in a while loop.  the examples I have found seem to assume it is known ahead of time how many items need to be processed.

specifically, I am reading from an external queue.  I currently process items one at a time until the queue is empty.  I wrote a wrapper function to handle connecting to the queue, pulling the next message, and deleting it when I am done.  ideally, I'd like to modify this wrapper function to take an additional argument (processes) to specify the number of messages to process simultaneously.

I've attached a script that captures what I am doing now.  unfortunately, the external queue object is not publicly accessible and I'm not quite sure how to set up a local object that would support testing.  any suggestions would be most welcome.


thanks,
Johan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140506/46777bda/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mp_test.py
Type: text/x-python
Size: 793 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20140506/46777bda/attachment.py>