multiprocessing.Pool, its queue, and pre-emption
ladasky at my-deja.com
Thu Sep 15 22:52:45 CEST 2011
Suppose that I have a multi-core computer with N CPU's, and I create a
multiprocessing.Pool in Python 2.6 with N-1 Processes.
(My rationale for choosing N-1 Processes was discussed here:
Then I use Pool.map_async() to break up a long-running task T1 into M
chunks, where M > 2N.
Suppose that I have a second, parallelizable, long-running task T2
that I want to address in REAL TIME when the need arises. Using Pool,
is there a way for me to insert the chunks of T2 at the HEAD of the
task queue, instead of at its TAIL?
I've been snooping around inside Pool, and I would guess that what I
want to do is to manipulate Pool._inqueue, which is a
multiprocessing.queues.SimpleQueue object. I haven't found any
documentation for SimpleQueue. It appears to have only the most
rudimentary of public methods -- put, get, and empty.
Reading further, I can see that multiprocessing.Queue (not the same as
SimpleQueue) has put_nowait(), which looks like what I need. Assuming
that I had access to put_nowait(), then I would need an option in
map_async() which would invoke put_nowait() rather than just plain
Does anyone know of an efficient way to accomplish this? I'm thinking
that a subclass of Pool which replaces the SimpleQueue with a full-
fledged Queue, and which overrides map_async() allowing for a no-wait
task, would be very useful... but maybe I'm re-inventing the wheel.
Thanks for your advice!
More information about the Python-list