It seems like the discussion so far is:
Victor: "I know people when people hear 'new API' they get scared and
think we're going to do a Python-3-like breaking transition, but don't
worry, we're never going to do that."
Nathaniel: "But then what does the new API add?"
Greg: "It lets us do a Python-3-like breaking transition!"
That is not what I am proposing but it seems too easy for people to misunderstand it as such. Sorry.
Between everything discussed across this thread I believe we have enough information to suggest that we can avoid an "everyone's afraid of a new 3" mistake by instead making a shim available with a proposed new API that works on top of existing Python VM(s) so that if we decide to drop the old API being public in the future, we could do so *without a breaking transition*.
I know that has always been my hope, especially if any new API is actually going to be more restrictive instead of broader.
Given that, I suggest not worrying about defining a new C API within the CPython project and release itself (yet).
+1 from me. Until we have a PEP outlining the actual proposed API I'm not ready to have it go into 'master'. Helping show the shape of the API by wrapping pre-existing APIs I think that's going to be the way to sell it.
Without an available benefit, little will use it (and given the function call overhead we want to isolate some concepts, we know it will perform worse on today's VMs).
That "top-5" module using it idea? Maintain forks (hooray for git) of whatever your definition of "top-5" projects is that use the new API instead of the CPython API. If you attempt this on things like NumPy, you may be shocked at the states (plural on purpose) of their extension module code. That holds true for a lot of popular modules.
Part of the point of this work is to demonstrate that non-incremental order of magnitude performance change can be had on a Python VM that only supports such an API can be done in its own fork of CPython, PyPy, VictorBikeshedPy, FbIsAfraidToReleaseANewGcVmPy, etc. implementation to help argue for figuring out a viable not-breaking-the-world transition plan to do such a C API change thing in CPython itself.
I think part of the challenge here (and I believe it has been brought up elsewhere) is no one knows what kind of API is necessary for some faster VM other than PyPy. To me, the only C API that would could potentially start working toward and promoting **today** is one which is stripped to its bare bones and worst mirrors Python syntax. For instance, I have seen PyTuple_GET_ITEM() brought up a couple of times. But that's not syntax in Python, so I wouldn't feel comfortable including that in a simplified API. You really only need attribute access and object calling to make object indexing work, although for simplicity I can see wanting to provide an indexing API.
But otherwise I think we are making assumptions here. For me, unless we are trying to trim the C API down to just what is syntactically supported in Python and in such a way that it hides all C-level details I feel like we are guessing at what's best for other VMs, both today and in the future, until they can tell us that e.g. tuple indexing is actually not a problem performance-wise.
And Just to be clear, I totally support coming up with a totally stripped-down C API as I have outlined above as that shouldn't be controversial for any VM that wants to have a C-level API.