[issue39877] Daemon thread is crashing in PyEval_RestoreThread() while the main thread is exiting the process

[STINNER Victor]

STINNER Victor <vstinner at python.org> added the comment:

I reopen the issue, my change introduced a *new* issue.

While trying to fix bpo-19466, work on PR 18848, I noticed that my commit eb4e2ae2b8486e8ee4249218b95d94a9f0cc513e introduced a race condition :-(

The problem is that while the main thread is executing Py_FinalizeEx(), daemon threads can be waiting in take_gil(). Py_FinalizeEx() calls _PyRuntimeState_SetFinalizing(runtime, tstate). Later, Py_FinalizeEx() executes arbitrary Python code in _PyImport_Cleanup(tstate) which releases the GIL to give a chance to other threads to execute:

            if (_Py_atomic_load_relaxed(&ceval->gil_drop_request)) {
                /* Give another thread a chance */
                if (_PyThreadState_Swap(&runtime->gilstate, NULL) != tstate) {
                    Py_FatalError("tstate mix-up");
                }
                drop_gil(ceval, tstate);

                /* Other threads may run now */

                /* Check if we should make a quick exit. */
                exit_thread_if_finalizing(tstate);

                take_gil(ceval, tstate);

                if (_PyThreadState_Swap(&runtime->gilstate, tstate) != NULL) {
                    Py_FatalError("orphan tstate");
                }
            }

At this point, one daemon thread manages to get the GIL: take_gil() completes... even if runtime->finalizing is not NULL. I expected that exit_thread_if_finalizing() would exit the thread, but exit_thread_if_finalizing() is now called *after* take_gil().

--

It's unclear to me when the GIL (the lock object) is destroyed and how we are supposed to destroy it, if an unknown number of daemon threads are waiting for it.

The GIL lock is created by PyEval_InitThreads(): create_gil() with MUTEX_INIT(gil->mutex).

The GIL lock is destroyed by _PyEval_FiniThreads(): destroy_gil() with MUTEX_FINI(gil->mutex).

----------
resolution: fixed -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39877>
_______________________________________