[pypy-issue] Issue #2434: Support pybind11 in conjunction with PyPy's cpyext: issue with heap types (pypy/pypy)

Wenzel Jakob issues-reply at bitbucket.org
Tue Nov 22 11:39:21 EST 2016

New issue 2434: Support pybind11 in conjunction with PyPy's cpyext: issue with heap types

Wenzel Jakob:


I'm the maintainer of [pybind11](https://github.com/pybind/pybind11), which is a modern library for creating bindings to C++ code (similar in spirit to Boost.Python). Hearing of the cpyext improvements in the latest release, I tried to see if pybind11 would work with the supported CPython API in PyPy. This would be huge, since it immediately make many projects available in PyPy.

The good news: I just had to make a few minor changes to get a few simple examples to compile. This is all in the WIP branch in pybind11 here: https://github.com/pybind/pybind11/pull/521

The bad news: I'm running into fundamental issues involving creation of types.

To reproduce:
$ git clone https://github.com/wjakob/pybind11
$ cd pybind11
$ cat > test.cpp <<EOF
#include <pybind11/pybind11.h>

namespace py = pybind11;

    py::module m("test");
    struct Test { };
    py::class_<Test>(m, "Test");

    return m.ptr();
$ g++ test.cpp -g3 -std=c++11 -I <path-to-pypy-include> -I include -undefined dynamic_lookup -shared -rdynamic -o test.so

Loading the module from PyPy leads to the following error:

$ pypy
Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 13 2016, 01:57:41)
[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> import cpyext
>>>> pyext.load_module('test.so', 'test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: global name 'pyext' is not defined
>>>> cpyext.load_module('test.so', 'test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: TypeError: can't set attributes on type object 'Test'

The error message is thrown when pybind11 basically calls (via CPython API)

setattr(Test, '__module__', 'test')

Digging into the PyPy source code reveals the following source of this error message:

def setdictvalue(self, space, name, w_value):
    if not self.is_heaptype():
        raise oefmt(space.w_TypeError,
                    "can't set attributes on type object '%N'", self)

So PyPy believes that ``Test`` is not a heap type. Which is confusing, because ``cpyext`` believes it is.

The CPython API calls boil down to:

PyHeapTypeObject *type = ((PyHeapTypeObject*) PyType_Type.tp_alloc(&PyType_Type, 0)

.. fill out remaining type fields like name, docs, ...

PyObject_SetAttrString(type->ht_type, "__module__", ...); // BOOM, this fails

Interestingly, the type instance returned by ``tp_alloc`` *already* has the heap type flag set, and it is still set after the ``PyType_Ready`` call. I can only assume that PyPy has its own heap type flag that somehow isn't synchronized with the types created by cpyext.

More information about the pypy-issue mailing list