multiprocessing.Pool, its queue, and pre-emption

John Ladasky ladasky at my-deja.com
Thu Sep 15 22:52:45 CEST 2011


Suppose that I have a multi-core computer with N CPU's, and I create a
multiprocessing.Pool in Python 2.6 with N-1 Processes.

(My rationale for choosing N-1 Processes was discussed here:
http://groups.google.com/group/comp.lang.python/browse_frm/thread/65ba3ccd4be8228c)

Then I use Pool.map_async() to break up a long-running task T1 into M
chunks, where M > 2N.

Suppose that I have a second, parallelizable, long-running task T2
that I want to address in REAL TIME when the need arises.  Using Pool,
is there a way for me to insert the chunks of T2 at the HEAD of the
task queue, instead of at its TAIL?

I've been snooping around inside Pool, and I would guess that what I
want to do is to manipulate Pool._inqueue, which is a
multiprocessing.queues.SimpleQueue object.  I haven't found any
documentation for SimpleQueue.  It appears to have only the most
rudimentary of public methods -- put, get, and empty.

Reading further, I can see that multiprocessing.Queue (not the same as
SimpleQueue) has put_nowait(), which looks like what I need.  Assuming
that I had access to put_nowait(), then I would need an option in
map_async() which would invoke put_nowait() rather than just plain
put().

Does anyone know of an efficient way to accomplish this?  I'm thinking
that a subclass of Pool which replaces the SimpleQueue with a full-
fledged Queue, and which overrides map_async() allowing for a no-wait
task, would be very useful... but maybe I'm re-inventing the wheel.
Thanks for your advice!



More information about the Python-list mailing list