[issue11271] concurrent.futures.ProcessPoolExecutor.map() doesn't batch function arguments by chunks

[Dan O'Reilly]

Dan O'Reilly added the comment:

I'm seeing an even larger difference between multiprocessing.Pool and ProcessPoolExecutor on my machine, with Python 3.4:

Starting multiproc...done in 2.160644769668579 s.
Starting futures...done in 67.953957319259644 s.
Starting futures "fixed"...done in 2.134932041168213 s.

I've updated the initial patch to address the comments Antoine made; the chunksize now defaults to 1, and itertools is used to chunk the input iterables, rather than building a list. Attached is an updated benchmark script:

Starting multiproc...done in 2.2295100688934326 s.
Starting futures...done in 68.5991039276123 s.
Starting futures "fixed" (no chunking)...done in 69.18992304801941 s.
Starting futures "fixed" (with chunking)...done in 2.352942705154419 s.

The new implementation of map has essentially identical performance to the original with chunksize=1, but it performs much better with a larger chunksize provided.

----------
nosy: +dan.oreilly
Added file: http://bugs.python.org/file36058/test_mult.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11271>
_______________________________________