[New-bugs-announce] [issue22292] pickle whichmodule RuntimeError

[attilio.dinisio]

New submission from attilio.dinisio:

= Error message =
pickle.dumps and pickle.dump generate the following error:

  File "/usr/lib/python3.4/pickle.py", line 283, in whichmodule
    for module_name, module in sys.modules.items():
RuntimeError: dictionary changed size during iteration


= Test case = 
The test case is as follows:

    import pickle
    import sys
    import numpy #version 1.8.1
    import scipy.signal #version 0.13.3
    pickle.dumps(numpy.cos)

Sometimes this results in the error

Traceback (most recent call last):
  File "whichmodule_bug.py", line 13, in <module>
    pickle.dumps(numpy.cos)
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 61, in _ufunc_reduce
    return _ufunc_reconstruct, (whichmodule(func, name), name)
  File "/usr/lib/python3.4/pickle.py", line 283, in whichmodule
    for module_name, module in sys.modules.items():
RuntimeError: dictionary changed size during iteration

NOTE: This is not necessarily a problem due to numpy and scipy packages. Explanation follows.


= Relevant code = 
The last version of pickle http://hg.python.org/cpython/file/f6f691ff27b9/Lib/pickle.py contains:
...

def _getattribute(obj, name, allow_qualname=False):
    dotted_path = name.split(".")
    if not allow_qualname and len(dotted_path) > 1:
        raise AttributeError("Can't get qualified attribute {!r} on {!r}; " +
                             "use protocols >= 4 to enable support"
                             .format(name, obj))
    for subpath in dotted_path:
        if subpath == '<locals>':
            raise AttributeError("Can't get local attribute {!r} on {!r}"
                                 .format(name, obj))
        try:
            obj = getattr(obj, subpath)
        except AttributeError:
            raise AttributeError("Can't get attribute {!r} on {!r}"
                                 .format(name, obj))
    return obj

def whichmodule(obj, name, allow_qualname=False):
    """Find the module an object belong to."""
    module_name = getattr(obj, '__module__', None)
    if module_name is not None:
        return module_name
    for module_name, module in sys.modules.items():
        if module_name == '__main__' or module is None:
            continue
        try:
            if _getattribute(module, name, allow_qualname) is obj:
                return module_name
        except AttributeError:
            pass
    return '__main__'
...

= History =
The iteration 

     for module_name, module in sys.modules.items():

in pickle.py was changed in 2007 to

     for name, module in list(sys.modules.items()):

by the BDFL [http://hg.python.org/cpython/rev/0d2dc3611e3b?revcount=800] to "Fix a bizarre error...", then it was reverted by [http://hg.python.org/cpython/rev/992ef855b3ed] in 2013. I hope the BDFL never discover this :-)


= Explanation = 
It seems that while iterating in sys.modules, getattr is called on an instance of scipy.lib.six.MovedModule, which triggers a change in sys.modules (due to an import). This result in the RuntimeError: dictionary changed size during iteration.
Freezing the list, with list(sys.modules.items()) resolves this particular issue.


= Notes on reproducing this error =
Reproducing my test case might be difficult for the following reason.
I installed scipy version 0.13.3 on Kubuntu 14.04 (sudo apt-get install python3-scipy). In the online  source code, module six.py version 1.2.0 is included [https://github.com/scipy/scipy/releases/tag/v0.13.3]. However in my system the module six.py that is actually used is 1.5.2, which probably has been installed independently from scipy. 
Finally, the new version of scipy, 0.14.0, doesn't use the class scipy.lib.six.MovedModule which triggers this error. Unfortunately this new version isn't available in Ubuntu 14.04.

----------
components: Library (Lib)
messages: 226005
nosy: attilio.dinisio, pitrou
priority: normal
severity: normal
status: open
title: pickle whichmodule RuntimeError
type: behavior
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22292>
_______________________________________