Get multiprocessing.Queue to do priorities
uuid
address_is at invalid.invalid
Sat May 9 14:23:25 EDT 2009
Scott David Daniels <Scott.Daniels at Acm.Org> wrote:
>
> ? "one producer, many consumers" ?
> What would the priority queue do? Choose a consumer?
Sorry, I should have provided a little more detail. There is one
producer thread, reading urls from multiple files and external input.
These urls have a certain priority, and are fed to multiple consumer
threads for fetching and further processing (XML parsing and such CPU
intensive stuff). Since the content of the urls is changing over time,
it is crucial to have a certain amount of control over the order in
which the requests occur. So, to answer the question:
The priority queue would make sure that out of a number of
asynchronously added items, those with a high priority are fetched first
by the worker threads. Sounds like a perfect case for a heap, if only I
could :)
More information about the Python-list
mailing list