[Python-ideas] Implement `Executor.map_as_completed`

Yannick Hold-Geoffroy yannickhold at gmail.com
Sun May 5 19:13:30 CEST 2013


Hello,

It may be interesting to highlight that multiprocessing handles this by
providing a imap_unordered
function<http://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.imap_unordered>
.

I provide an interface compatible to the futures
(PEP-3148<http://www.python.org/dev/peps/pep-3148/>)
in my parallel framework (SCOOP <http://scoop.googlecode.com/>) and I also
feel the need for a simple way to iterate through results as they are
completed.
I understand that the *map()* function should be as generic as possible and
finer grained concurrency should be explicitly handled by the user, but as
Mr. Rachum said, a shortcut between the finest granularity (user have to
submit futures manually) and generic serial-like map could be of use (It
would help write DRY code since this is a really common use case).

I am not necessarily for the creation of a new function implicitly doing an
unordered iteration, though. Since futures.map returns an iterator, I think
we should provide users with a function that wraps the futures.map and
allow users to explicitly call it for unordered iteration. This could be
very intuitive -- Like allowing futures.as_completed to accept
futures.map() output. This could be done using the object-oriented
philosophy (by keeping some internal state linking the map result to their
related future) or functional philosophy (by returning an object behaving
like the result with the addition of an identifier to the future).
Internally, the function could either implement a callback returning the
value to the user as soon as results are available or call as_completed on
their related futures.

Either way (new function or new wrapper) would greatly simplify some usual
use cases of concurrent or parallel programs using or inspired by PEP-3148.

Have a nice day,
Yannick Hold
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130505/3e08b90a/attachment.html>


More information about the Python-ideas mailing list