[issue20297] concurrent.futures.as_completed() installs waiters for already completed Futures

[Glenn Langford]

New submission from Glenn Langford:

as_completed() calls _create_and_install_waiters() over all given Futures, even if some (or all) of the Futures have already completed.

The following fragment of the as_completed code (from 3.3.3):

with _AcquireFutures(fs):
        finished = set(
                f for f in fs
                if f._state in [CANCELLED_AND_NOTIFIED, FINISHED])
        pending = set(fs) - finished
        waiter = _create_and_install_waiters(fs, _AS_COMPLETED)

installs waiters into Futures contained in fs, rather than pending. 

A more efficient approach might be to only install waiters on the pending subset if necessary. This would be faster and would also result in less dwell time with all of the Future conditions locked via the _AcquireFutures context manager.

Also: waiters are appended with the Future condition lock acquired, but are removed (at the conclusion of as_completed) without the _AcquireFutures condition lock. Is this correct?

----------
components: Library (Lib)
messages: 208418
nosy: glangford
priority: normal
severity: normal
status: open
title: concurrent.futures.as_completed() installs waiters for already completed Futures
type: performance
versions: Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20297>
_______________________________________