[New-bugs-announce] [issue37127] Handling pending calls during runtime finalization may cause problems.

[Eric Snow]

New submission from Eric Snow <ericsnowcurrently at gmail.com>:

In Python/lifecycle.c (Py_FinalizeEx) we call _Py_FinishPendingCalls(), right after we stop all non-daemon Python threads but before we've actually started finalizing the runtime state.  That call looks for any remaining pending calls (for the main interpreter) and runs them.  There's some evidence of a bug there.

In bpo-33608 I moved the pending calls to per-interpreter state.  We saw failures (sometimes sporadic) on a few buildbots (e.g. FreeBSD) during runtime finalization.  However, nearly all of the buildbots were fine, so it may be a question of architecture or slow hardware.  See bpo-33608 for details on the failures.

There are a number of possibilities, but it's been tricky reproducing the problem in order to investigate.  Here are some theories:

* daemon threads (a known weak point in runtime finalization) block pending calls from happening until some time after portions of the runtime have already been cleaned up
* there's a race that causes the pending calls machinery to get caught in some sort infinite loop (e.g. a pending call fails and gets re-queued)
* a corner case in the pending calls logic that triggers only during finalization

Here are some other points to consider:

* do we have the same problem during subinterpreter finalization (Py_EndInterpreter() rather than runtime finalization)?
* perhaps the problem extends beyond finalization, but the conditions are more likely there
* the change for bpo-33608 could have introduced the bug rather that exposing an existing one

----------
assignee: eric.snow
components: Interpreter Core
messages: 344202
nosy: eric.snow
priority: normal
severity: normal
stage: test needed
status: open
title: Handling pending calls during runtime finalization may cause problems.
type: crash

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37127>
_______________________________________