[pypy-issue] Issue #2436: Support pybind11 in conjunction with PyPy's cpyext: Crash in PyDict_Next (pypy/pypy)

Wenzel Jakob issues-reply at bitbucket.org
Wed Nov 23 19:30:06 EST 2016

New issue 2436: Support pybind11 in conjunction with PyPy's cpyext: Crash in PyDict_Next

Wenzel Jakob:


following the fix in Issue #2434, I was able to get basic types & methods to export properly in [pybind11](https://github.com/pybind/pybind11). Also, the entire test suite compiles now (it segfaults when actually running it, but still..). Yay! :)

Looking into the test suite crashes, it turns out that the culprit is a call to ``PyDict_Next()`` to traverse the contents of a newly created type. Pybind11 uses this function to iterate over the dictionary of a type when binding C++ enumerations in Python (to do some basic postprocessing of the enumeration entries).

The CPython API calls boil down to this:

PyObject *dict = ((PyTypeObject *) ... /* Pointer to newly created type */)->tp_dict;
PyObject *key, *value;
ssize_t pos = 0;

while (PyDict_Next(dict, &pos, &key, &value)) {
    /* Do something -- Never gets here because PyDict_Next crashes */

This works fine in Python 2.7 and 3.x but crashes in PyPy.

What's weird is that ``dict`` is a perfectly good dictionary even in PyPy. I can for instance turn it into a string and print it before the ``PyDict_Next()`` call, getting

{'__init__': <unbound method pybind11_tests.EMode.__init__>, '__new__': <builtin_function_or_method object at 0x0000000104754250>, '__dict__': <getset_descriptor object at 0x00000001019724d8>, '__weakref__': <getset_descriptor object at 0x000000010191b068>, '__doc__': None, '__module__': 'pybind11_tests', '__repr__': <unbound method pybind11_tests.EMode.__repr__>, '__int__': <unbound method pybind11_tests.EMode.__int__>, '__eq__': <unbound method pybind11_tests.EMode.__eq__>, '__ne__': <unbound method pybind11_tests.EMode.__ne__>, '__hash__': <unbound method pybind11_tests.EMode.__hash__>, '__getstate__': <unbound method pybind11_tests.EMode.__getstate__>, '__setstate__': <unbound method pybind11_tests.EMode.__setstate__>, 'EFirstMode': EMode.EFirstMode, 'ESecondMode': EMode.ESecondMode}

but ``PyDict_Next()`` still crashes. 

LLDB reports the following backtrace:

  * frame #0: 0x0000000000000000
    frame #1: 0x000000010062c0a4 libpypy-c.dylib`pypy_g_PyDict_Next + 244
    frame #2: 0x00000001005f479b libpypy-c.dylib`pypy_g_wrapper_second_level__star_4_4 + 347
    frame #3: 0x0000000105d8b0e0 pybind11_tests.pypy-41.so`pybind11::enum_<MyEnum>::export_values(this=0x00007fff5fbfe580) + 80 at pybind11.h:1260

which is bizarre. If I read that correctly, PyPy seems to have done a jump to NULL.

How to reproduce on your end:

$ git clone https://github.com/wjakob/pybind11
$ cd pybind11
$ cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-pypy>
$ make
$ pypy -m pip install pytest
$ pypy -m pytest

Thank you,

More information about the pypy-issue mailing list