[New-bugs-announce] [issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration

[Markus Mohrhard]

New submission from Markus Mohrhard <markusm at dug.com>:

We have hit an issue in the pickle module where the code throws an exception in a threaded environment:

The interesting piece of the backtrace is:

  File "/xxx/1004060/lib/python3.7/site-packages/numpy/core/__init__.py", line 130, in _ufunc_reduce
    return _ufunc_reconstruct, (whichmodule(func, name), name)
  File "/xxx/lib/python3.7/pickle.py", line 309, in whichmodule
    for module_name, module in list(sys.modules.items()):
RuntimeError: dictionary changed size during iteration

I tried to find a code path that would explain how the dict could be changed while the list is created but have not been able to find a code path that releases the GIL.

The executable is using many threads with imports happening in random threads and a custom class loader but we already make sure that the class loader is always holding the GIL.

The issue happens quite rarely (maybe once every one thousand's execution) so I don't have a reproducer right now.

----------
components: Extension Modules
messages: 366762
nosy: Markus Mohrhard
priority: normal
severity: normal
status: open
title: list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration
type: behavior
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40327>
_______________________________________