PyCodeObject incompatibility
Hello! I'm trying to build a low-level C API extension of cPython to be used in PyPy. The extension extensively uses some fields of PyCodeObject ( https://github.com/python/cpython/blob/f87e616af038ee8963185e11b96841c81e8ef...): co_firstlineno, co_stacksize, co_consts and so on. However, it seems these fields are not defined in the similar structure of the PyPy implementation: https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.8/pypy/module/cpyext/i.... Is it possible to get values of these fields using PyPy C API somehow?
Hi, On Thu, 10 Feb 2022 at 14:03, Dmitry Kovner <dmitryk@lightrun.com> wrote:
Hello! I'm trying to build a low-level C API extension of cPython to be used in PyPy. The extension extensively uses some fields of PyCodeObject (https://github.com/python/cpython/blob/f87e616af038ee8963185e11b96841c81e8ef...): co_firstlineno, co_stacksize, co_consts and so on. However, it seems these fields are not defined in the similar structure of the PyPy implementation: https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.8/pypy/module/cpyext/i.... Is it possible to get values of these fields using PyPy C API somehow?
Yes, it's a limitation of PyPy not to expose these fields. If you want to write portable code that always works, the simplest is to call `PyObject_GetAttrString(code, "__consts__")` etc. Remember that you need to call `Py_DECREF()` on the result at some point. A bientôt, Armin.
Hi, again! First of all, thanks for the fast reply! The answer helped me a lot. However, during the porting of the extension to the awesome PyPy, I've got more questions: 1. The extension uses opcodes from cPython's opcode.h header file. ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) There is no such file in the implementation of C API in PyPy. What could you recommend as the best alternative? 2. The extension uses some fields of PyThreadState structure. For example, its frame field ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...). Is it right to use PyObject_GetAttrString() to access that field in PyPy C API? 3. There is no function PyFrame_FastToLocals ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) in the C API of PyPy. Are there any alternatives? 4. Functions PyEval_SetTrace(), PyEval_SetProfile(), PyFrame_GetLineNumber() are defined only in stubs.py in PyPy. ( https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.7/pypy/module/cpyext/s...) Is there a way to get that information in PyPy except patching of its source code? 5. Are there any analogues of all PyTrace_* constants ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) in the PyPy C API? 6. There is no function PyFrame_Check() ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) in the PyPy C API. Are there any alternatives? I'm really sorry for the long question! I hope the list of the problems described above is full enough to port the extension and I will have no more questions. :) Best regards, Dmitrii чт, 10 февр. 2022 г. в 23:07, Armin Rigo <armin.rigo@gmail.com>:
On 15/2/22 05:15, Dmitry Kovner wrote:
Even if PyPy could expose the interfaces you desire (and it would be a lot of work to do that), you would lose out on speed since the C-API on PyPy is much much slower than pure python. If you really want to port your extension, you might want to rethink what it is doing and refactor it to do so in a PyPy-friendly way: pure python + CFFI where needed to interface to libraries that expose C interfaces. The C-API is a layer we bolted on top of PyPy to support popular libraries like Cython and NumPy, it is not meant for projects that probe CPython internals. Matti
Hi Dmitry, I can only reiterate, we would need some information about what that library is actually for to give more useful information at this point. As Matti said, most of these APIs cannot be usefully implemented in PyPy. most jitted functions do not *have* a filled frame object at all in PyPy. so if you want to access the frames, you need to leave the jit-produced machine code, which is very slow. Cheers, CF On 15.02.22 04:15, Dmitry Kovner wrote:
Hi Dmitry, those fields are quite implementation-dependant, it's not clear to me that their use can be just carried over from CPython in all cases. Could you maybe tell us which exception module uses them or at least what you are doing with them? But yes, as Armin write, accessing the the .co_* attributes with PyObject_GetAttrString is an approach! Cheers, CF On 10.02.22 07:01, Dmitry Kovner wrote:
Hi, On Thu, 10 Feb 2022 at 14:20, Carl Friedrich Bolz-Tereick <cfbolz@gmx.de> wrote:
are doing with them? But yes, as Armin write, accessing the the .co_* attributes with PyObject_GetAttrString is an approach!
Oops, sorry. Python 3 renamed various attributes to use the double-underscore convention, like on function objects, but skipped code objects for some reason. So I meant `PyObject_GetAttrString(code, "co_consts")`. Armin
Hi, On Thu, 10 Feb 2022 at 14:03, Dmitry Kovner <dmitryk@lightrun.com> wrote:
Hello! I'm trying to build a low-level C API extension of cPython to be used in PyPy. The extension extensively uses some fields of PyCodeObject (https://github.com/python/cpython/blob/f87e616af038ee8963185e11b96841c81e8ef...): co_firstlineno, co_stacksize, co_consts and so on. However, it seems these fields are not defined in the similar structure of the PyPy implementation: https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.8/pypy/module/cpyext/i.... Is it possible to get values of these fields using PyPy C API somehow?
Yes, it's a limitation of PyPy not to expose these fields. If you want to write portable code that always works, the simplest is to call `PyObject_GetAttrString(code, "__consts__")` etc. Remember that you need to call `Py_DECREF()` on the result at some point. A bientôt, Armin.
Hi, again! First of all, thanks for the fast reply! The answer helped me a lot. However, during the porting of the extension to the awesome PyPy, I've got more questions: 1. The extension uses opcodes from cPython's opcode.h header file. ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) There is no such file in the implementation of C API in PyPy. What could you recommend as the best alternative? 2. The extension uses some fields of PyThreadState structure. For example, its frame field ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...). Is it right to use PyObject_GetAttrString() to access that field in PyPy C API? 3. There is no function PyFrame_FastToLocals ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) in the C API of PyPy. Are there any alternatives? 4. Functions PyEval_SetTrace(), PyEval_SetProfile(), PyFrame_GetLineNumber() are defined only in stubs.py in PyPy. ( https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.7/pypy/module/cpyext/s...) Is there a way to get that information in PyPy except patching of its source code? 5. Are there any analogues of all PyTrace_* constants ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) in the PyPy C API? 6. There is no function PyFrame_Check() ( https://github.com/python/cpython/blob/8a84aef0123bd8c13cf81fbc3b5f6d45f96c2...) in the PyPy C API. Are there any alternatives? I'm really sorry for the long question! I hope the list of the problems described above is full enough to port the extension and I will have no more questions. :) Best regards, Dmitrii чт, 10 февр. 2022 г. в 23:07, Armin Rigo <armin.rigo@gmail.com>:
On 15/2/22 05:15, Dmitry Kovner wrote:
Even if PyPy could expose the interfaces you desire (and it would be a lot of work to do that), you would lose out on speed since the C-API on PyPy is much much slower than pure python. If you really want to port your extension, you might want to rethink what it is doing and refactor it to do so in a PyPy-friendly way: pure python + CFFI where needed to interface to libraries that expose C interfaces. The C-API is a layer we bolted on top of PyPy to support popular libraries like Cython and NumPy, it is not meant for projects that probe CPython internals. Matti
Hi Dmitry, I can only reiterate, we would need some information about what that library is actually for to give more useful information at this point. As Matti said, most of these APIs cannot be usefully implemented in PyPy. most jitted functions do not *have* a filled frame object at all in PyPy. so if you want to access the frames, you need to leave the jit-produced machine code, which is very slow. Cheers, CF On 15.02.22 04:15, Dmitry Kovner wrote:
Hi Dmitry, those fields are quite implementation-dependant, it's not clear to me that their use can be just carried over from CPython in all cases. Could you maybe tell us which exception module uses them or at least what you are doing with them? But yes, as Armin write, accessing the the .co_* attributes with PyObject_GetAttrString is an approach! Cheers, CF On 10.02.22 07:01, Dmitry Kovner wrote:
Hi, On Thu, 10 Feb 2022 at 14:20, Carl Friedrich Bolz-Tereick <cfbolz@gmx.de> wrote:
are doing with them? But yes, as Armin write, accessing the the .co_* attributes with PyObject_GetAttrString is an approach!
Oops, sorry. Python 3 renamed various attributes to use the double-underscore convention, like on function objects, but skipped code objects for some reason. So I meant `PyObject_GetAttrString(code, "co_consts")`. Armin
participants (4)
-
Armin Rigo -
Carl Friedrich Bolz-Tereick -
Dmitry Kovner -
Matti Picus