Get multiprocessing.Queue to do priorities
uuid
M8R-gqmgb4 at mailinator.com
Sat May 9 13:00:54 EDT 2009
I just read up and it seems that no matter the approach, it's futile to
use multiprocessing.Queue, since there is a bug that prevents true
FIFO. (http://bugs.python.org/issue4999)
Any recommendation on an alternate way to build a priority queue to use
with a "one producer, many consumers" type multiprocessing setup would
be welcomed!
:|
On 2009-05-09 18:42:34 +0200, uuid <M8R-gqmgb4 at mailinator.com> said:
> Hello,
> I was wondering whether there was a way to make multiprocessing.Queue
> behave in a priority queue-like fashion. Subclassing with heappush and
> heappop for put and get doesn't work the old way (multiprocessing.Queue
> seems to use different data structures than Queue.Queue?)
>
> Could one create a heapq within the producer as a proxy, and then feed
> a proper queue from that? Does anyone have an idea on how to deal with
> the queue item flow control? (Since all of the sorting has to happen
> within the heapq, it should only pass items to the real Queue if it's
> empty?)
>
> Thanks in advance!
More information about the Python-list
mailing list