Pool module -- does map pre-assign tasks to processes?
Aahz
aahz at pythoncraft.com
Mon Oct 5 14:14:35 EDT 2009
In article <a6593d46-0f0e-4ca7-8481-c53b18218f57 at o13g2000vbl.googlegroups.com>,
Luca <luca.de.alfaro at gmail.com> wrote:
>I would like to use the Pool module, but my tasks take sometimes
>unpredictably different time to complete. The simplest way to write
>the code would be to put all task descriptions in an array, then call
>
>p = Pool(8)
>p.map(f, a)
>
>But I don't want to preassign process 0 with elements a[0], a[8], a
>[16], process 1 with elements a[1], a[9], a[17], and so forth.
>Rather, I would like all tasks to be put into a queue, and I would
>like the processes to each time grab the next task to be done, and do
>it. This would ensure fairly equal loading.
>
>My question is: does the map() method of Pool pre-assign which task
>gets done by each process, or is this done at runtime, on a get-first-
>task-to-be-done basis?
You can probably get what you want by setting chunksize to 1.
--
Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/
"....Normal is what cuts off your sixth finger and your tail..." --Siobhan
More information about the Python-list
mailing list