Steve Dower wrote:
On a policy level, we don't make changes that would break users of the C API. Because we can't track everyone who's using it, we have to assume that everything is used and any change will cause breakage.
To make sure it's possible to keep developing CPython, we declare parts of the API off limits (typically by prepending them with an underscore). If you use these, and you break, we're sorry but we aren't going to fix it.
This line of discussion is basically saying that we would designate a broader section of the API that is off limits, most likely the parts that are only useful for increased performance (rather than increased functionality). We would then specifically include the Cython team/volunteers in discussions about how to manage changes to these parts of the API to avoid breaking them, and possibly do simultaneous releases to account for changes so that their users have more time to rebuild.
Effectively, when we change our APIs, we would break everyone except Cython because we've worked with them to avoid the breakage. Anyone else using it has to make their own effort to follow CPython development and detect any breakage themselves (just like today).
So probably the part you're missing is where we would give ourselves permission to break more APIs in a release, while simultaneously encouraging people to use Cython as an isolation layer from those breaks.
The encouraging part is not working for me :-) And seriously, my gut tells me, we're split at 50/50 here. People usually write C for a reason and Cython is not. For, let's say, half of the cases that's fine, speeding up inner loops and all that, which not touching the C level at all. The other half wants to solve different issues.
I think, it does not serve well as a policy for CPython. Since we're talking hypotheticals right now, if Cython vanishes tomorrow, we're kind of left empty handed. Such kind of a runtime, if considered part of the compatibility "promise", should be provided by the core itself, no? A good way to test that promise (or other implications like performance) might also be to rewrite the standard library extensions in Cython and see where it leads.
I personally see myself using the python-provided runtime (types, methods, GC), out of convenience (it's there, so why not use it). The vision of the future outlined here can easily lead to backing off from that and rebuilding all those things and really only keep touchpoints with python when it comes to interfacing with python itself. It's probably even desirable that way. But definitely more work (for an extension author).
As a closing word, I don't mind either way. IOW I'm not complaining. I'm just putting more opinion from the "outside" into the ring. Thanks for listening :-)