Hi Victor,
Thanks for putting work into this. I support the idea of slowly evolving the C API. It must be done carefully so as to not unnecessarily break 3rd party extensions. Changes must be made for well founded reasons and not just because we think it makes a "cleaner" API. I believe you are following those principles.
One aspect of the API that could be improved is memory management for PyObjects. The current API is quite a mess and for no good reason except legacy, IMHO. The original API design allowed extension types to use their own memory allocator. E.g. they could call their own malloc()/free() implemention and the rest of the CPython runtime would handle that. One consequence is that Py_DECREF() cannot call PyObject_Free() but instead has to call tp_dealloc(). There was supposed to be multiple layers of allocators, PyMem vs PyObject, but since the layering was not enforced, we ended up with a bunch of aliases to the same underlying function.
Perhaps there are a few cases when the flexibility to use a custom object allocator is useful. I think in practice it is very rare than an extension needs to manage memory itself. To achieve something similar, allow a PyObject to have a reference to some externally managed resource and then the tp_del method would take care of freeing it. IMHO, the Python runtime should be in charge of allocating and freeing PyObject memory.
I believe fixing this issue is not tricky, just tedious. The biggest hurdle might be dealing with statically allocated objects. IMHO, they should go away and there should only be heap allocated PyObjects (created and freed by calling CPython API functions). That change would affect most extensions, unfortunately.
Another place for improvement is that the C API is unnecessarily large. E.g. we don't really need PyList_GetItem(), PyTuple_GetItem(), and PyObject_GetItem(). Every extra API is a potential leak of implementation details and a burden for alternative VMs. Maybe we should introduce something like WIN32_LEAN_AND_MEAN that hides all the extra stuff. The Py_LIMITED_API define doesn't really mean the same thing since it tries to give ABI compatibility. It would make sense to cooperate with the HPy project on deciding what parts are unnecessary. Things like Cython might still want to use the larger API, to extract every bit of performance. The vast majority of C extensions don't require that.
One final comment: I think even if we manage to cleanup the API and make it friendly for other Python implementations, there is going to be a fair amount of overhead. If you look at other "managed runtimes" that just seems unavoidable (e.g. Java, CLR, V8, etc). You want to design the API so that you maximize the amount of useful work done with each API call. Using something like PyList_GET_ITEM() to iterate over a list is not a good pattern. So keep in mind that an extension API is going to have some overhead.
Regards,
Neil