PyCXX and PyPy status
I'm the maintainer of the PyCXX C++ extension interface to Python C API. PyCXX source code is a SVN repo at: https://svn.code.sf.net/p/cxx/code/trunk/CXX I have been working to get PyCXX to build and run against PyPy C API. I am happy to have PyPy specific code in PyCXX where necessary. Using PyPy 0.6.0 on macOS I managed to build a .so that crashed when imported. The reason is that the PyPy's C API does not allow tuples to be created. The pattern in CPython is like this: explicit Tuple (const Sequence& s) { sequence_index_type limit( sequence_index_type( s.length() ) ); set(PyTuple_New (limit), true); validate(); for(sequence_index_type i=0; i < limit; i++) { if(PyTuple_SetItem (ptr(), i, new_reference_to(s[i])) == -1) { ifPyErrorThrowCxxException(); } } } The PyTuple_SetItem() always fails. Its a must have to create tuples. How do you expect a tuple to be created? I have a build of the tip from the PyPy HG repo. This has regressed from the 0.6.0 release: g++ -c -g -Wall -fPIC -fexceptions -frtti -I. -ISrc -I/home/barry/tmpdir/usession-default-barry/build/pypy-qqq/include -DNDEBUG -oobj/simple.obj Demo/Python2/simple.cxx In file included from ./CXX/Objects.hxx:40, from Demo/Python2/simple.cxx:15: ./CXX/Python2/Objects.hxx: In member function ‘void Py::MapBase<T>::delItem(const string&)’: ./CXX/Python2/Objects.hxx:3041:17: error: there are no arguments to ‘PyMapping_DelItemString’ that depend on a template parameter, so a declaration of ‘PyMapping_DelItemString’ must be available [-fpermissive] if (PyMapping_DelItemString (ptr(), const_cast<char*>(s.c_str())) == -1) ^~~~~~~~~~~~~~~~~~~~~~~ ./CXX/Python2/Objects.hxx:3041:17: note: (if you use ‘-fpermissive’, G++ will accept your code, but allowing the use of an undeclared name is deprecated) ./CXX/Python2/Objects.hxx: In member function ‘void Py::MapBase<T>::delItem(const Py::Object&)’: ./CXX/Python2/Objects.hxx:3049:17: error: there are no arguments to ‘PyMapping_DelItem’ that depend on a template parameter, so a declaration of ‘PyMapping_DelItem’ must be available [-fpermissive] if (PyMapping_DelItem (ptr(), *s) == -1) ^~~~~~~~~~~~~~~~~ It seems that PyMapping_DelItem and PyMapping_DelString have been removed since 0.6.0. Can they be restored? Oh and where does the magic "44" come from that I had to use in the <extension>.pypy-44.so file name? Barry
Hi Barry, On Tue, 2 Oct 2018 at 21:09, Barry Scott <barry@barrys-emacs.org> wrote:
Using PyPy 0.6.0 on macOS I managed to build a .so that crashed when imported.
I will assume that you mean PyPy2 6.0.0, and not PyPy3 nor the version 0.6.0.
The reason is that the PyPy's C API does not allow tuples to be created.
You cannot change tuple objects after they "escaped" to become general PyPy objects. You can only call PyTuple_SetItem() on a tuple object that exists as a "PyObject *" but not yet as a PyPy object. That means, mostly, you should only call PyTuple_SetItem() on the fresh result of PyTuple_New() before the "PyObject *" is sent somewhere else. In your example code, maybe the set() function makes the tuple object escape in this way. Note that escaping tuples before they are fully initialized can also sometimes create potential crashes on CPython.
if (PyMapping_DelItemString (ptr(), const_cast<char*>(s.c_str())) == -1) if (PyMapping_DelItem (ptr(), *s) == -1)
I don't know how it ever worked, because PyMapping_DelItem and PyMapping_DelItemString are not implemented, neither right now nor at the time of release 6.0.0. If you are using these two functions, please open an issue (https://bitbucket.org/pypy/pypy/issues?status=new&status=open) and we'll implement them. A bientôt, Armin.
Le 02/10/18 à 21:55, Armin Rigo a écrit :
I just implemented these in https://bitbucket.org/pypy/pypy/commits/a5735c0b1edd6e5e16dc7015facf2131fed6...
Thanks I'm compiling again. I see that the .so will not be found unless I use the .pypy-41.so suffix. Where does the 41 come from? I'm getting further. I have changed the details of how I create tuples to make progress. I will go back and figure out the impact on my users later. I'm now getting a lot further. Module has completed init. Old style classes seem to be working. But new style classes are not working. When I make a new style class its type has the following tp_flags: (gdb) p/x table->tp_flags $4 = 0x201eb But when the instance comes back to me they are: (gdb) p/x self->ob_type->tp_flags $11 = 0x1208 Surely the flags should not have been changed? Since Py_TPFLAGS_BASETYPE has gone from 1 to 0. I crash because I assume the wrong type of class and access a pointer that is not there. Thoughts? Barry
Hi, On Sun, 7 Oct 2018 at 18:05, Barry Scott <barry@barrys-emacs.org> wrote:
Some flags change in CPython too. The change you're seeing might be correct, or it might not be---although I agree that the bits in the two values are very different. But it's hard for us to know at this point without knowing what you are doing exactly. You may be using only the standard parts of the CPython C API, or instead be using completely internal details. Can you try to extract a small CPython extension module as C code, containing the usual kind of PyTypeObject definitions and calls to some PyXxx() functions, and which behaves differently on CPython and on PyPy? That would help us understand the problem. A bientôt, Armin.
To my knowledge I'm not using internal APIs.
Where is your test code for the this feature? I could review/modify that to help find a difference that may explain the problem. If you have no test coverage for new-style-classes I would rather not write that code for you. Instead I can document how to build the PyCXX test code to run against PyPy for you. It is not difficult. Barry
A bientôt,
Armin.
Hi Barry, On Sat, 27 Oct 2018 at 16:22, Barry Scott <barry@barrys-emacs.org> wrote:
Where is your test code for the this feature? I could review/modify that to help find a difference that may explain the problem.
You are saying that your use case fails, but we don't know which part of cpyext is causing the failure. I can point to the whole tests of cpyext (they are in pypy/module/cpyext/test/; new-style classes tests are in test_typeobject). But that seems not useful. What I'm asking for is any way for us to reproduce the problem. It can be step-by-step instructions about building PyCXX (like you suggested); or it can be a smaller .c use case, which might help to motivate us to look at it. It doesn't have to be specially prepared or integrated with the cpyext tests---we first need to figure out what the real problem is, before we can write a relevant unit test. A bientôt, Armin.
Hi Barry, On Tue, 2 Oct 2018 at 21:09, Barry Scott <barry@barrys-emacs.org> wrote:
Using PyPy 0.6.0 on macOS I managed to build a .so that crashed when imported.
I will assume that you mean PyPy2 6.0.0, and not PyPy3 nor the version 0.6.0.
The reason is that the PyPy's C API does not allow tuples to be created.
You cannot change tuple objects after they "escaped" to become general PyPy objects. You can only call PyTuple_SetItem() on a tuple object that exists as a "PyObject *" but not yet as a PyPy object. That means, mostly, you should only call PyTuple_SetItem() on the fresh result of PyTuple_New() before the "PyObject *" is sent somewhere else. In your example code, maybe the set() function makes the tuple object escape in this way. Note that escaping tuples before they are fully initialized can also sometimes create potential crashes on CPython.
if (PyMapping_DelItemString (ptr(), const_cast<char*>(s.c_str())) == -1) if (PyMapping_DelItem (ptr(), *s) == -1)
I don't know how it ever worked, because PyMapping_DelItem and PyMapping_DelItemString are not implemented, neither right now nor at the time of release 6.0.0. If you are using these two functions, please open an issue (https://bitbucket.org/pypy/pypy/issues?status=new&status=open) and we'll implement them. A bientôt, Armin.
Le 02/10/18 à 21:55, Armin Rigo a écrit :
I just implemented these in https://bitbucket.org/pypy/pypy/commits/a5735c0b1edd6e5e16dc7015facf2131fed6...
Thanks I'm compiling again. I see that the .so will not be found unless I use the .pypy-41.so suffix. Where does the 41 come from? I'm getting further. I have changed the details of how I create tuples to make progress. I will go back and figure out the impact on my users later. I'm now getting a lot further. Module has completed init. Old style classes seem to be working. But new style classes are not working. When I make a new style class its type has the following tp_flags: (gdb) p/x table->tp_flags $4 = 0x201eb But when the instance comes back to me they are: (gdb) p/x self->ob_type->tp_flags $11 = 0x1208 Surely the flags should not have been changed? Since Py_TPFLAGS_BASETYPE has gone from 1 to 0. I crash because I assume the wrong type of class and access a pointer that is not there. Thoughts? Barry
Hi, On Sun, 7 Oct 2018 at 18:05, Barry Scott <barry@barrys-emacs.org> wrote:
Some flags change in CPython too. The change you're seeing might be correct, or it might not be---although I agree that the bits in the two values are very different. But it's hard for us to know at this point without knowing what you are doing exactly. You may be using only the standard parts of the CPython C API, or instead be using completely internal details. Can you try to extract a small CPython extension module as C code, containing the usual kind of PyTypeObject definitions and calls to some PyXxx() functions, and which behaves differently on CPython and on PyPy? That would help us understand the problem. A bientôt, Armin.
To my knowledge I'm not using internal APIs.
Where is your test code for the this feature? I could review/modify that to help find a difference that may explain the problem. If you have no test coverage for new-style-classes I would rather not write that code for you. Instead I can document how to build the PyCXX test code to run against PyPy for you. It is not difficult. Barry
A bientôt,
Armin.
Hi Barry, On Sat, 27 Oct 2018 at 16:22, Barry Scott <barry@barrys-emacs.org> wrote:
Where is your test code for the this feature? I could review/modify that to help find a difference that may explain the problem.
You are saying that your use case fails, but we don't know which part of cpyext is causing the failure. I can point to the whole tests of cpyext (they are in pypy/module/cpyext/test/; new-style classes tests are in test_typeobject). But that seems not useful. What I'm asking for is any way for us to reproduce the problem. It can be step-by-step instructions about building PyCXX (like you suggested); or it can be a smaller .c use case, which might help to motivate us to look at it. It doesn't have to be specially prepared or integrated with the cpyext tests---we first need to figure out what the real problem is, before we can write a relevant unit test. A bientôt, Armin.
participants (3)
-
Armin Rigo -
Barry Scott -
Ronan Lamy