On 2019-03-03, Nick Coghlan wrote:
Given the long history of the existing CPython C API, I think any new handle-based design is still going to have to support the invariant that "c_handle_X == c_handle_Y" <=> "python_object_X is python_object_Y".
What is not clear to me yet is how difficult it would be for other Python VMs to implement a handle API that gives that invariant. Based on some dicussion with Armin, it sounds like PyPy takes a significant performance hit to do that. If your GC can move objects, how do you efficiently implement that invariant?
I guess different people have different ideas on the goals of a new API. To me, the goals are quite similar to what lead to Java's Native Interface. I want an API so extensions modules (native code, in Java speak) work efficiently with multiple Python VMs. The API for extensions should not unduly constrain the implementation choices of the VM. One critical point is that the VM should be able to use an object moving GC.
For the new API to displace the existing one, there are some further requirements:
must be easy to convert existing extension modules to new API
must be possible to use converted extension with older CPython versions (provide shim layer)
new API should not be lower performance or less capable than existing API
existing API is still available for modules that are not converted
Anything else will be far too error prone (Consider that JNI has *never* worked that way, and people still get it wrong repeatedly based on assumption about how object identity should work in C/C++. How much worse would the problem be for us given that Python's C API *has* worked that way for the past ~30 years?)
You might be correct but I wonder if the problem is so bad for Python. You already have to be very careful when comparing objects by id. E.g. interned strings compare by id but non-interned ones don't. Or, small integers compare by id but not if they are larger. So, maybe extension code would actually get better if we more strongly enforce the rule that you can't assume handles can be compared like that.
I worry that if we decide up front that the new API must work like the existing API in these ways, we are going to force ourselves into a design that is too CPython specific. Then, there will be no benefit to other VMs with the new API vs the existing one.
Regards,
Neil