[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Petr Viktorin]

On 11/19/18 12:14 PM, Victor Stinner wrote:
> To design a new C API, I see 3 options:
> 
> (1) add more functions to the existing Py_LIMITED_API
> (2) "fork" the current public C API: remove functions and hide as much
> implementation details as possible
> (3) write a new C API from scratch, based on the current C API.
> Something like #define newcapi_Object_GetItem PyObject_GetItem"?
> Sorry, but "#undef <private_function>" doesn't work. Only very few
> functions are defined using "#define ...".
> 
> I dislike (1) because it's too far from what is currently used in
> practice. Moreover, I failed to find anyone who can explain me how the
> C API is used in the wild, which functions are important or not, what
> is the C API, etc.

One big, complex project that now uses the limited API is PySide. They 
do some workarounds, but the limited API works. Here's a writeup of the 
troubles they have with it: 
https://github.com/pyside/pyside2-setup/blob/5.11/sources/shiboken2/libshiboken/pep384impl_doc.rst

> I propose (2). We control how much changes we do at each milestone,
> and we start from the maximum compatibility with current C API. Each
> change can be discussed and experimented to define what is the C API,
> what we want, etc. I'm working on this approach for 1 year, that's why
> many discussions popped up around specific changes :-)

I hope the new C API will be improvements (and clarifications) of the 
stable ABI, rather than a completely new thing.
My ideal would be that Python 4.0 would keep the same API (with 
questionable things emulated & deprecated), but break *ABI*. The "new C 
API" would become that new stable ABI -- and this time it'd be something 
we'd really want to support, without reservations.

One thing that did not work with the stable ABI was that it's "opt-out"; 
I think we can agree that a new one must be "opt-in" from the start.
I'd also like the "new API" to be a *strict subset* of the stable ABI: 
if a new function needs to be added, it should be added to both.

> Some people recently proposed (3) on python-dev. I dislike this option
> because it starts by breaking the backward compatibility. It looks
> like (1), but worse. The goal and the implementation are unclear to
> me.
> 
> --
> 
> Replacing PyDict_GetItem() (specialized call) with PyObject_Dict()
> (generic API) is not part of my short term plan. I wrote it in the
> roadmap, but as I wrote before, each change should be discusssed,
> experimented, benchmarked, etc.
> 
> Victor
> Le lun. 19 nov. 2018 à 12:02, M.-A. Lemburg <mal at egenix.com> a écrit :
>>
>> On 19.11.2018 11:53, Antoine Pitrou wrote:
>>> On Mon, 19 Nov 2018 11:28:46 +0100
>>> Victor Stinner <vstinner at redhat.com> wrote:
>>>> Python internals rely on internals to implement further optimizations,
>>>> than modifying an "immutable" tuple, bytes or str object, because you
>>>> can do that at the C level. But I'm not sure that I would like 3rd
>>>> party extensions to rely on such things.
>>>
>>> I'm not even talking about *modifying* tuples or str objects, I'm
>>> talking about *accessing* their value without going through an abstract
>>> API that does slot lookups, indirect function calls and object unboxing.
>>>
>>> For example, people may need a fast way to access the UTF-8
>>> representation of a unicode object.  Without making indirect function
>>> calls, and ideally without making a copy of the data either.  How do
>>> you do that using the generic C API?
>>
>> Something else you need to consider is creating instances of
>> types, e.g. a tuple. In C you will have to be able to put
>> values into the data structure before it is passed outside
>> the function in order to build the tuple.
>>
>> If you remove this possibility to have to copy data all the
>> time, losing the advantages of having a rich C API.
>>   --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Experts (#1, Nov 19 2018)
>>>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>>> Python Database Interfaces ...           http://products.egenix.com/
>>>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
>> ________________________________________________________________________
>>
>> ::: We implement business ideas - efficiently in both time and costs :::
>>
>>     eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>      D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>             Registered at Amtsgericht Duesseldorf: HRB 46611
>>                 http://www.egenix.com/company/contact/
>>                        http://www.malemburg.com/
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com
>