[Python-checkins] bpo-36607: Eliminate RuntimeError raised by asyncio.all_tasks() (GH-13971)

[Miss Islington (bot)]

https://github.com/python/cpython/commit/5d1d4e3179ffd4d2d72462d870cf86dcc11450ce
commit: 5d1d4e3179ffd4d2d72462d870cf86dcc11450ce
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-06-11T09:01:20-07:00
summary:

bpo-36607: Eliminate RuntimeError raised by asyncio.all_tasks() (GH-13971)


If internal tasks weak set is changed by another thread during iteration.

https://bugs.python.org/issue36607
(cherry picked from commit 65aa64fae89a24491aae84ba0329eb8f3c68c389)

Co-authored-by: Andrew Svetlov <andrew.svetlov at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-06-11-13-52-04.bpo-36607.5_mJkQ.rst
M Lib/asyncio/tasks.py

diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py
index 402c6e26a79b..c82b95e3beea 100644
--- a/Lib/asyncio/tasks.py
+++ b/Lib/asyncio/tasks.py
@@ -35,9 +35,22 @@ def all_tasks(loop=None):
     """Return a set of all tasks for the loop."""
     if loop is None:
         loop = events.get_running_loop()
-    # NB: set(_all_tasks) is required to protect
-    # from https://bugs.python.org/issue34970 bug
-    return {t for t in list(_all_tasks)
+    # Looping over a WeakSet (_all_tasks) isn't safe as it can be updated from another
+    # thread while we do so. Therefore we cast it to list prior to filtering. The list
+    # cast itself requires iteration, so we repeat it several times ignoring
+    # RuntimeErrors (which are not very likely to occur). See issues 34970 and 36607 for
+    # details.
+    i = 0
+    while True:
+        try:
+            tasks = list(_all_tasks)
+        except RuntimeError:
+            i += 1
+            if i >= 1000:
+                raise
+        else:
+            break
+    return {t for t in tasks
             if futures._get_loop(t) is loop and not t.done()}
 
 
@@ -47,9 +60,22 @@ def _all_tasks_compat(loop=None):
     # method.
     if loop is None:
         loop = events.get_event_loop()
-    # NB: set(_all_tasks) is required to protect
-    # from https://bugs.python.org/issue34970 bug
-    return {t for t in list(_all_tasks) if futures._get_loop(t) is loop}
+    # Looping over a WeakSet (_all_tasks) isn't safe as it can be updated from another
+    # thread while we do so. Therefore we cast it to list prior to filtering. The list
+    # cast itself requires iteration, so we repeat it several times ignoring
+    # RuntimeErrors (which are not very likely to occur). See issues 34970 and 36607 for
+    # details.
+    i = 0
+    while True:
+        try:
+            tasks = list(_all_tasks)
+        except RuntimeError:
+            i += 1
+            if i >= 1000:
+                raise
+        else:
+            break
+    return {t for t in tasks if futures._get_loop(t) is loop}
 
 
 class Task(futures._PyFuture):  # Inherit Python Task implementation
diff --git a/Misc/NEWS.d/next/Library/2019-06-11-13-52-04.bpo-36607.5_mJkQ.rst b/Misc/NEWS.d/next/Library/2019-06-11-13-52-04.bpo-36607.5_mJkQ.rst
new file mode 100644
index 000000000000..337c8e2668c4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-06-11-13-52-04.bpo-36607.5_mJkQ.rst
@@ -0,0 +1,2 @@
+Eliminate :exc:`RuntimeError` raised by :func:`asyncio.all_tasks()` if
+internal tasks weak set is changed by another thread during iteration.