[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

Nathaniel Smith njs at pobox.com
Tue Nov 13 22:17:31 EST 2018


On Sun, Nov 11, 2018 at 3:19 PM, Victor Stinner <vstinner at redhat.com> wrote:
> I'm not sure yet how far we should go towards a perfect API which
> doesn't leak everything. We have to move slowly, and make sure that we
> don't break major C extensions. We need to write tools to fully
> automate the conversion. If it's not possible, maybe the whole project
> will fail.

This is why I'm nervous about adding this directly to CPython. If
we're just talking about adding a few new API calls to replace old
ones that are awkward to use, then that's fine, that's not very risky.
But if you're talking about a large project that makes fundamental
changes in the C API (e.g., disallowing pointer dereferences, like
tagged pointers do), then yeah, there's a very large risk that that
might fail.

>> If so, then would it make more sense to develop this as an actual>> separate abstraction layer? That would have the huge advantage that it
>> could be distributed and versioned separately from CPython, different
>> packages could use different versions of the abstraction layer, PyPy
>> isn't forced to immediately add a bunch of new APIs...
>
> I didn't investigate this option. But I expect that you will have to
> write a full new API using a different prefix than "Py_". Otherwise,
> I'm not sure how you want to handle PyTuple_GET_ITEM() as a macro on
> one side (Include/tupleobject.h) and PyTuple_GET_ITEM() on the other
> side (hypotetical_new_api.h).
>
> Would it mean to duplicate all functions to get a different prefix?
>
> If you keep the "Py_" prefix, what I would like to ensure is that some
> functions are no longer accessible. How you remove
> PySequence_Fast_GET_ITEM() for example?
>
> For me, it seems simpler to modify CPython headers than starting on
> something new. It seems simpler to choose the proper level of
> compatibility. I start from an API 100% compatible (the current C
> API), and decide what is changed and how.

It may be simpler, but it's hugely more risky. Once you add something
to CPython, you can't take it back again without a huge amount of
work. You said above that the whole project might fail. But if it's in
CPython, failure is not acceptable! The whole problem you're trying to
solve is that the C API is too big, but your proposed solution starts
by making it bigger, so if your project fails then it makes the
problem even bigger...

I don't know if making it a separate project is the best approach or
not, it was just an idea :-). But it would have the huge benefit that
you can actually experiment and try things out without committing to
supporting them forever.

And I don't know the best answer to all your questions above, that's
what experimenting is for :-). But it certainly is technically
possible to make a new API that shares a common subset with the old
API, e.g.:

/* NewPython.h */
#include <Python.h>
#define PyTuple_GET_ITEM PyTuple_Get_Item
#undef PySequence_Fast_GET_ITEM

-n

-- 
Nathaniel J. Smith -- https://vorpus.org


More information about the Python-Dev mailing list