How can we make Python's C-API more friendly for cpyext?
Hello, I'm figuring out a strategy for the future of Python's C-API, and as one of the goals I'd like to make it friendly for non-C languages. I'd like to ask you for any suggestions, pain points or comments on how to better do that. Would you like to help? I understand PyPy solved a lot of these issues and won't get too much benefit, but I'd like to improve things for future projects. I'm also aware of HPy, a long-term solution. I'd like to meet in the middle. I started a [Google doc](https://docs.google.com/document/d/1fDfF7JanOEyQ9awhKE87-Ajp3H4k7qXL3cmyzPMo...) to collect notes. If you don't mind Google, please feel free to comment there, but I'll be happy to discuss here as well. If you have a link to a document/rant, I'd love to go through it. In particular, I'd like to ask if any of the following would help, or would have helped in the past: - Regularity in strong/borrowed references (always returning strong references and taking borrowed ones) - Regularity in error handling (all functions should have a dedicated return value, an exception should be set set iff that value is returned) - Avoiding preprocessor macros - Avoiding bit fields - Avoiding enums - Exposing API/ABI information in a structured format, rather than C headers Is anything else missing from this list? Thank you for your time!
On 29/4/22 20:20, encukou@gmail.com wrote:
Thanks for reaching out. One person's opinion to get the discussion going: The first two points about regularity would help. Preprocessor macros are problematic where we need to reproduce the exact defines used by the "configure" front end to compilation. I don't think the bit fields, or enums are too hard for PyPy to handle, we use cpyparser to parse the C syntax and I think those are handled correctly. I am not sure what you mean by "structured format", for us C structs are fine, some XML or JSON constructs would probably be harder to use. One thing not on the list is assuming that objects can be resurrected during deallocation. The PyPy garbage collector cannot resurrect objects. This is less a problem for pure-python code than for C extensions that implement complicated functionality inside tp_dealloc, or when calling PyObject_CallFinalizerFromDealloc (which is not supported on PyPy at all). Another thing would be better code examples as snippets of code to test the interfaces that are part of the stable API. For instance, I have been unable to write such a sample for PyCFunction_New [0]. Having such snippets as part of a PEP to add a new interface would make it much easier to implement new functionality. The VectorCall functionality would have been much easier to emulate if the spec was completely described by code snippets. Matti [0] https://foss.heptapod.net/pypy/pypy/-/issues/3601
On 30. 04. 22 22:08, Matti Picus wrote:
Thanks for the reply! Apologies for the delay -- right after I asked, “life happened” (literally 👶). I'm back now :)
Could you elaborate on this one? Is this about macros in pyconfig.h, or pypy's configure?
Oof, I'm afraid this is a bit out of my reach. I'll need to call this out of scope for this thread :(
Are there other functions PyPy doesn't/can't support? We might want to discourage them and mark them as CPython-specific.
Roger, will try to do better next time. But feel free to reach out to PEP authors and ask for such examples. You could even open issues after a PEP is accepted, since the snippets you need would make good tests for CPython.
On 29/4/22 20:20, encukou@gmail.com wrote:
Thanks for reaching out. One person's opinion to get the discussion going: The first two points about regularity would help. Preprocessor macros are problematic where we need to reproduce the exact defines used by the "configure" front end to compilation. I don't think the bit fields, or enums are too hard for PyPy to handle, we use cpyparser to parse the C syntax and I think those are handled correctly. I am not sure what you mean by "structured format", for us C structs are fine, some XML or JSON constructs would probably be harder to use. One thing not on the list is assuming that objects can be resurrected during deallocation. The PyPy garbage collector cannot resurrect objects. This is less a problem for pure-python code than for C extensions that implement complicated functionality inside tp_dealloc, or when calling PyObject_CallFinalizerFromDealloc (which is not supported on PyPy at all). Another thing would be better code examples as snippets of code to test the interfaces that are part of the stable API. For instance, I have been unable to write such a sample for PyCFunction_New [0]. Having such snippets as part of a PEP to add a new interface would make it much easier to implement new functionality. The VectorCall functionality would have been much easier to emulate if the spec was completely described by code snippets. Matti [0] https://foss.heptapod.net/pypy/pypy/-/issues/3601
On 30. 04. 22 22:08, Matti Picus wrote:
Thanks for the reply! Apologies for the delay -- right after I asked, “life happened” (literally 👶). I'm back now :)
Could you elaborate on this one? Is this about macros in pyconfig.h, or pypy's configure?
Oof, I'm afraid this is a bit out of my reach. I'll need to call this out of scope for this thread :(
Are there other functions PyPy doesn't/can't support? We might want to discourage them and mark them as CPython-specific.
Roger, will try to do better next time. But feel free to reach out to PEP authors and ask for such examples. You could even open issues after a PEP is accepted, since the snippets you need would make good tests for CPython.
participants (3)
-
encukou＠gmail.com -
Matti Picus -
Petr Viktorin