[Python-ideas] PEP: Hide implementation details in the C API

Ronald Oussoren ronaldoussoren at mac.com
Thu Jul 13 12:30:37 EDT 2017

> On 13 Jul 2017, at 13:46, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2017-07-12 20:51 GMT+02:00 Brett Cannon <brett at python.org>:
>> I also think the motivation doesn't have to be performance but simply
>> cleaning up how we expose our C APIs to users as shown by the fact we have
>> messed up the stable API by making it opt-out instead of opt-in.
> It's hard to sell a "cleanup" to users with no carrot :-) Did someone
> remind trying to sell the "Python 3 cleanup"? :-)

But then there should actually be a carrot ;-). 

Just declaring the contents of object definitions private in the documentation could also help, especially when adding preprocessor guards to enable access to those definitions. Consulting adults etc… 

>> Yeah, that's also what I initially thought. Use a cascading hierarchy so
>> that people know they should put anything as high up as possible to minimize
>> its exposure.
> Yeah, maybe we can do that.
> I have to make my own experiment to make sure that #include doesn't
> leak symbols by mistakes and that it's still possible to use optimized
> macros or functions in builtin modules.

Avoiding symbol leaks with a cascading hierarchy should be easy enough,
some care may be needed to be able to override definitions in the “more private”
headers, especially when making current macros available as functions in the
more public headers. Although it could be considered to just remove macros
like PyTuple_GET_ITEM from the most public layer.

>> I think Victor has long-term plans to try and hide the struct details at a
>> higher-level and so that would make macros a bad thing. But ignoring the
>> specific Py_INCREF/DECREF example, switching to functions does buy us the
>> ability to actually change the function implementations between Python
>> versions compared to having to worry about what a macro used to do (which is
>> a possibility with the stable ABI).
> I think that my PEP is currently badly written :-)
> In fact, the idea is just to make the stable ABI usable :-) Instead of
> hiding structures *and* remove macros, my idea is just to hide
> structures but still provides macros... as functions. Basically, it
> will be the same API, but usable on more various implementations of
> Python.

It might be better to push users towards tools like ctypes and cffi, the latter especially
is tuned to work both with CPython and PyPy and appears to gain momentum. That
won’t work for everything, but could work for a large subset of extensions.


More information about the Python-ideas mailing list