[New-bugs-announce] [issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
report at bugs.python.org
Mon Feb 21 16:04:49 CET 2011
New submission from Tobias Brink <tobias.brink at gmail.com>:
I tested the new concurrent.futures.ProcessPoolExecutor.map() in 3.2 with the is_prime() function from the documentation example. This was significantly slower than using multiprocessing.Pool.map(). Quick look at the source showed that multiprocessing sends the iterable in chunks to the worker process while futures sends always only one entry of the iterable to the worker.
Functions like is_prime() which finish relatively fast make the communication overhead (at least I guess that is the culprit) very big in comparison.
Attached is a file which demonstrates the problem and a quick workaround. The workaround uses the chunk idea from multiprocessing. The problem is that it requires the iterables passed to map() to have a length and be indexable with a slice. I believe this limitation could be worked around.
components: Library (Lib)
title: concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
versions: Python 3.2
Added file: http://bugs.python.org/file20825/map_comparison.py
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce