[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Victor Stinner]

Le sam. 10 nov. 2018 à 04:02, Nathaniel Smith <njs at pobox.com> a écrit :
> So is it fair to say that your plan is that CPython will always use
> the current ("old") API internally, and the "new" API will be
> essentially an abstraction layer, that's designed to let people write
> C extensions that target the old API, while also being flexible enough
> to target PyPy and other "new different Python runtimes"?

What we call is "Python C API" is not really an API. I mean, nobody
tried to sit down and think about a proper API to access Python from
C. We happened is that we had an implementation of Python written in C
and it was cool to just "expose everything". By everything, I mean
"everything". It's hard to find a secret CPython feature not exposed
or leaked in the C API.

The problem that I'm trying to solve to "fix the C API" to hide
implementation details, with one constraint: don't create a new
"Python 4". I don't want to break the backward compatibility without
having a slow and smooth transition plan. The "old" and new C API must
be supported in parallel, at least in the standard CPython,
/usr/bin/python3.

Writing a new API from scratch is nice, but it's harder to moving all
existing C extensions from the "old" C API to a new one.

Replacing macros with functions has little impact on backward
compatibility. Most C extensions should still work if macros become
functions.

I'm not sure yet how far we should go towards a perfect API which
doesn't leak everything. We have to move slowly, and make sure that we
don't break major C extensions. We need to write tools to fully
automate the conversion. If it's not possible, maybe the whole project
will fail.

I'm looking for a practical solutions based on the existing C API and
the existing CPython code base.

> If so, then would it make more sense to develop this as an actual
> separate abstraction layer? That would have the huge advantage that it
> could be distributed and versioned separately from CPython, different
> packages could use different versions of the abstraction layer, PyPy
> isn't forced to immediately add a bunch of new APIs...

I didn't investigate this option. But I expect that you will have to
write a full new API using a different prefix than "Py_". Otherwise,
I'm not sure how you want to handle PyTuple_GET_ITEM() as a macro on
one side (Include/tupleobject.h) and PyTuple_GET_ITEM() on the other
side (hypotetical_new_api.h).

Would it mean to duplicate all functions to get a different prefix?

If you keep the "Py_" prefix, what I would like to ensure is that some
functions are no longer accessible. How you remove
PySequence_Fast_GET_ITEM() for example?

For me, it seems simpler to modify CPython headers than starting on
something new. It seems simpler to choose the proper level of
compatibility. I start from an API 100% compatible (the current C
API), and decide what is changed and how.

Victor