[issue20566] asyncio as_completed() thrashes adding and removing callbacks

[Glenn Langford]

New submission from Glenn Langford:

In asyncio, tasks.py as_completed() appears to trigger adding and removing callbacks multiple times for the pending set of futures, each time a single future completes. 

For example, to wait for 5 futures which complete at different times:
- as_completed() uses _wait()
- _wait() will add 5 callbacks, one for each future
- when one future completes, a callback is triggered and all 5 callbacks are removed; this is because _wait() was called with FIRST_COMPLETED
- for the 4 remaining futures - 4 callbacks have to be added back again
- when the next future completes, the 4 callbacks are removed
etc…

The worst case is if as_completed() is called to wait for all of a larger number of futures, which complete at different times. For example, with 100 futures worst case, ~10,000 callback adds and removes would be performed.

(I am very new to the asyncio code, so I don't have a patch to offer at this point).

----------
components: Library (Lib)
messages: 210679
nosy: glangford, gvanrossum, haypo, pitrou
priority: normal
severity: normal
status: open
title: asyncio as_completed() thrashes adding and removing callbacks
type: performance
versions: Python 3.4, Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20566>
_______________________________________