[New-bugs-announce] [issue28812] Deadlock between GIL and pystate head_mutex.

[INADA Naoki]

New submission from INADA Naoki:

While investigating #28673, I found another deadlock relating to finalization and threading.

deadlocked thread (holding gil / waiting head_mutex):
#0  0x00007f35dd400a07 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x5557e6d3eeb0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem at entry=0x5557e6d3eeb0, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f35dd400ab4 in __new_sem_wait_slow (sem=0x5557e6d3eeb0, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f35dd400b5a in __new_sem_wait (sem=sem at entry=0x5557e6d3eeb0) at sem_wait.c:29
#4  0x00005557e64ec01b in PyThread_acquire_lock_timed (lock=0x5557e6d3eeb0, microseconds=-1, intr_flag=intr_flag at entry=0) at Python/thread_pthread.h:352
#5  0x00005557e64ec148 in PyThread_acquire_lock (lock=<optimized out>, waitflag=waitflag at entry=1) at Python/thread_pthread.h:556
#6  0x00005557e64d8dbb in tstate_delete_common (tstate=tstate at entry=0x7f35cc002a80) at Python/pystate.c:426
#7  0x00005557e64d9e8d in PyThreadState_DeleteCurrent () at Python/pystate.c:462
#8  0x00005557e662d339 in t_bootstrap (boot_raw=0x7f35dc186628) at ./Modules/_threadmodule.c:1027
#9  0x00007f35dd3f870a in start_thread (arg=0x7f35c3fff700) at pthread_create.c:333
#10 0x00007f35dca220af in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

main thread (waiting gil / holding head_mutex):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00005557e65f1758 in PyCOND_TIMEDWAIT (us=5000, mut=0x5557e69cc860 <gil_mutex>, cond=0x5557e69cc8a0 <gil_cond>) at Python/condvar.h:103
#2  take_gil (tstate=tstate at entry=0x5557e6d3f840) at Python/ceval_gil.h:224
#3  0x00005557e65f3b14 in _PyEval_EvalFrameDefault (f=Frame 0x7f35d0001878, for file finish-deadlock.py, line 7, in __del__ (self=<C at remote 0x7f35dc1b6878>), throwflag=<optimized out>)
    at Python/ceval.c:1140
#4  0x00005557e65f1dcf in PyEval_EvalFrameEx (f=f at entry=Frame 0x7f35d0001878, for file finish-deadlock.py, line 7, in __del__ (self=<C at remote 0x7f35dc1b6878>), throwflag=throwflag at entry=0)
    at Python/ceval.c:718
#5  0x00005557e65f1eab in _PyFunction_FastCall (co=co at entry=0x7f35dc50eac0, args=0x7fff1a1e2878, args at entry=0x7fff1a1e2870, nargs=nargs at entry=1, 
    globals=globals at entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='finish-deadlock.py') at remote 0x7f35dd6c00c0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f35dd791c58>, '__cached__': None, 'sys': <module at remote 0x7f35dd7b9858>, 'time': <module at remote 0x7f35dc4ea6d8>, 'threading': <module at remote 0x7f35dc4ea8d8>, 'C': <type at remote 0x5557e6dc96c8>, 'worker': <function at remote 0x7f35dc4d53f0>, '_': 4, 't': <Thread(_target=<function at remote 0x7f35dc4d53f0>, _name='Thread-11', _args=(), _kwargs={}, _daemonic=True, _ident=139869249451776, _tstate_lock=<_thread.lock at remote 0x7f35dc4739e0>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7f35dc42be58>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f35dc42be58>, release=<built-in method release of _thread.lock object at remote 0x7f35dc42be58>, _waiters=<collections.deque at remote 0x7f35dc16d5b0>) at remote 0x7f35dc191538>,...(truncated)) at Python/ceval.c:4881
#6  0x00005557e65feb04 in _PyFunction_FastCallDict (func=func at entry=<function at remote 0x7f35dc11f788>, args=args at entry=0x7fff1a1e2870, nargs=nargs at entry=1, kwargs=kwargs at entry=0x0)
    at Python/ceval.c:4983
#7  0x00005557e6500386 in _PyObject_FastCallDict (func=func at entry=<function at remote 0x7f35dc11f788>, args=args at entry=0x7fff1a1e2870, nargs=nargs at entry=1, kwargs=kwargs at entry=0x0)
    at Objects/abstract.c:2301
#8  0x00005557e6500644 in _PyObject_Call_Prepend (func=<function at remote 0x7f35dc11f788>, obj=obj at entry=<C at remote 0x7f35dc1b6878>, args=args at entry=(), kwargs=kwargs at entry=0x0)
    at Objects/abstract.c:2364
#9  0x00005557e651c2e4 in method_call (method=method at entry=<method at remote 0x7f35dc521758>, args=args at entry=(), kwargs=kwargs at entry=0x0) at Objects/classobject.c:317
#10 0x00005557e6500247 in _PyObject_FastCallDict (func=func at entry=<method at remote 0x7f35dc521758>, args=args at entry=0x0, nargs=nargs at entry=0, kwargs=kwargs at entry=0x0) at Objects/abstract.c:2322
#11 0x00005557e65f34a2 in PyEval_CallObjectWithKeywords (func=func at entry=<method at remote 0x7f35dc521758>, args=args at entry=0x0, kwargs=kwargs at entry=0x0) at Python/ceval.c:4705
#12 0x00005557e6577f0d in slot_tp_finalize (self=<optimized out>) at Objects/typeobject.c:6416
#13 0x00005557e655a07c in PyObject_CallFinalizer (self=self at entry=<C at remote 0x7f35dc1b6878>) at Objects/object.c:297
#14 0x00005557e655adb7 in PyObject_CallFinalizerFromDealloc (self=self at entry=<C at remote 0x7f35dc1b6878>) at Objects/object.c:314
#15 0x00005557e656cfde in subtype_dealloc (self=self at entry=<C at remote 0x7f35dc1b6878>) at Objects/typeobject.c:1151
#16 0x00005557e655ae75 in _Py_Dealloc (op=<C at remote 0x7f35dc1b6878>) at Objects/object.c:1786
#17 0x00005557e65312c5 in frame_dealloc (f=f at entry=Frame 0x7f35c40026a8, for file finish-deadlock.py, line 22, in worker ()) at Objects/frameobject.c:423
#18 0x00005557e655ae75 in _Py_Dealloc (op=Frame 0x7f35c40026a8, for file finish-deadlock.py, line 22, in worker ()) at Objects/object.c:1786
#19 0x00005557e64e88c1 in tb_dealloc (tb=tb at entry=0x7f35dc193f38) at Python/traceback.c:55
#20 0x00005557e655ae75 in _Py_Dealloc (op=<traceback at remote 0x7f35dc193f38>) at Objects/object.c:1786
#21 0x00005557e64d98f3 in PyThreadState_Clear (tstate=tstate at entry=0x5557e6e307f0) at Python/pystate.c:403
#22 0x00005557e64d99cf in PyInterpreterState_Clear (interp=interp at entry=0x5557e6d3f390) at Python/pystate.c:119
#23 0x00005557e64d7b0f in Py_FinalizeEx () at Python/pylifecycle.c:672
#24 0x00005557e64eed2f in Py_Main (argc=2, argv=0x5557e6d3e010) at Modules/main.c:800
#25 0x00005557e64d3ad5 in main (argc=2, argv=0x7fff1a1e2d78) at ./Programs/python.c:69

----------
components: Interpreter Core
files: finish-deadlock.py
messages: 281816
nosy: inada.naoki
priority: normal
severity: normal
status: open
title: Deadlock between GIL and pystate head_mutex.
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7
Added file: http://bugs.python.org/file45660/finish-deadlock.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28812>
_______________________________________