[Python-checkins] bpo-20526: Fix PyThreadState_Clear(): don't decref frame (GH-19120) (GH-19136) (GH-19137)

[Victor Stinner]

https://github.com/python/cpython/commit/d1c09896c3b91d0ad7e3a14fabecde268f70dac7
commit: d1c09896c3b91d0ad7e3a14fabecde268f70dac7
branch: 3.7
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2020-03-24T17:56:01+01:00
summary:

bpo-20526: Fix PyThreadState_Clear(): don't decref frame (GH-19120) (GH-19136) (GH-19137)

PyThreadState.frame is a borrowed reference, not a strong reference:
PyThreadState_Clear() must not call Py_CLEAR(tstate->frame).

Remove test_threading.test_warnings_at_exit(): we cannot warranty
that the Python thread state of daemon threads is cleared in a
reliable way during Python shutdown.

(cherry picked from commit 5804f878e779712e803be927ca8a6df389d82cdf)
(cherry picked from commit e97c8b0688bc62959ced477d842fcd37992ef649)

files:
A Misc/NEWS.d/next/Core and Builtins/2020-03-23-18-08-34.bpo-20526.NHNZIv.rst
M Include/pystate.h
M Python/pystate.c

diff --git a/Include/pystate.h b/Include/pystate.h
index f16ffb8fd2aef..819c1427a745e 100644
--- a/Include/pystate.h
+++ b/Include/pystate.h
@@ -216,6 +216,7 @@ typedef struct _ts {
     struct _ts *next;
     PyInterpreterState *interp;
 
+    /* Borrowed reference to the current frame (it can be NULL) */
     struct _frame *frame;
     int recursion_depth;
     char overflowed; /* The stack has overflowed. Allow 50 more calls
diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-03-23-18-08-34.bpo-20526.NHNZIv.rst b/Misc/NEWS.d/next/Core and Builtins/2020-03-23-18-08-34.bpo-20526.NHNZIv.rst
new file mode 100644
index 0000000000000..c808b7608c61e
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2020-03-23-18-08-34.bpo-20526.NHNZIv.rst	
@@ -0,0 +1,3 @@
+Fix :c:func:`PyThreadState_Clear()`. ``PyThreadState.frame`` is a borrowed
+reference, not a strong reference: ``PyThreadState_Clear()`` must not call
+``Py_CLEAR(tstate->frame)``.
diff --git a/Python/pystate.c b/Python/pystate.c
index ebfe6ce65de03..604e0279148cb 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -566,11 +566,19 @@ _PyState_ClearModules(void)
 void
 PyThreadState_Clear(PyThreadState *tstate)
 {
-    if (Py_VerboseFlag && tstate->frame != NULL)
+    if (Py_VerboseFlag && tstate->frame != NULL) {
+        /* bpo-20526: After the main thread calls
+           _PyRuntimeState_SetFinalizing() in Py_FinalizeEx(), threads must
+           exit when trying to take the GIL. If a thread exit in the middle of
+           _PyEval_EvalFrameDefault(), tstate->frame is not reset to its
+           previous value. It is more likely with daemon threads, but it can
+           happen with regular threads if threading._shutdown() fails
+           (ex: interrupted by CTRL+C). */
         fprintf(stderr,
           "PyThreadState_Clear: warning: thread still has a frame\n");
+    }
 
-    Py_CLEAR(tstate->frame);
+    /* Don't clear tstate->frame: it is a borrowed reference */
 
     Py_CLEAR(tstate->dict);
     Py_CLEAR(tstate->async_exc);