[issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument

[Tobias Brink]

New submission from Tobias Brink <tobias.brink at gmail.com>:

I tested the new concurrent.futures.ProcessPoolExecutor.map() in 3.2 with the is_prime() function from the documentation example. This was significantly slower than using multiprocessing.Pool.map(). Quick look at the source showed that multiprocessing sends the iterable in chunks to the worker process while futures sends always only one entry of the iterable to the worker.

Functions like is_prime() which finish relatively fast make the communication overhead (at least I guess that is the culprit) very big in comparison.

Attached is a file which demonstrates the problem and a quick workaround.  The workaround uses the chunk idea from multiprocessing.  The problem is that it requires the iterables passed to map() to have a length and be indexable with a slice.  I believe this limitation could be worked around.

----------
components: Library (Lib)
files: map_comparison.py
messages: 128963
nosy: tbrink
priority: normal
severity: normal
status: open
title: concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
type: performance
versions: Python 3.2
Added file: http://bugs.python.org/file20825/map_comparison.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11271>
_______________________________________