[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Neil Schemenauer]

On 2018-11-19, Antoine Pitrou wrote:
> There are important use cases for the C API where it is desired to have
> fast type-specific access to Python objects such as tuples, ints,
> strings, etc.  This is relied upon by modules such as _json and _pickle,
> and third-party extensions as well.

Thank you for pointing this out.  The feedback from Stefan on what
Cython would like (e.g. more access to functions that are currently
"internal") is useful too.  Keeping our dreams tied to reality
is important. ;-P

It seems to me that we can't "have our cake and eat it too". I.e. on
the one hand hide CPython implementation internals but on the other
hand allow extensions that want to take advantage of those internals
to provide the best performance.

Maybe we could have a multiple levels of API:

A) maximum portability (Py_LIMITED_API)

B) source portability (non-stable ABI, inlined functions)

C) portability but poor performance on non-CPython VMs
   (PySequence_Fast_ITEMS, borrowed refs, etc)

D) non-portability, CPython specific (access to more internals like
   Stefan was asking for).  The extension would have to be
   re-implemented on each VM or provide a pure Python
   alternative.

I think it would be nice if the extension module could explicitly
choose which level of API it wants to use.

It would be interesting to do a census on what extensions are out
there.  If they mostly fall into wanting level "C" then I think this
API overhaul is not going to work out too well.  Level C is mostly
what we have now.  No point in putting the effort into A and B if no
one will use them.