[Python-Dev] PEP 384: Defining a Stable ABI

Nick Coghlan ncoghlan at gmail.com
Mon May 25 23:04:58 CEST 2009


M.-A. Lemburg wrote:
> Now, with the PEP, I have a feeling that the Python C-API
> will in effect be limited to what's in the PEP's idea of
> a usable ABI and open up the non-inluded public C-APIs
> to the same rate of change as the private APIs.

Not really - before this PEP it was already fairly easy to write an
extension that was source-level compatible with multiple versions of
Python (depending on exactly what you wanted to do, of course).

However, it is essentially impossible to make an extension that is
binary level compatible with multiple versions.

With the defined stable ABI in place, each extension module author will
be able to make a choice:
- choose binary compatibility by limiting themselves to the stable ABI
and be able to provide a single binary that will still work with later
versions of Py3k
- stick with source compatibility and continue to provide new binaries
for each version of Python

> An optional cross-version ABI would certainly be a good thing.
> 
> Limiting the Python C-API would be counterproductive.

I don't think anyone would disagree with that. A discussion on C-API sig
would certainly be a good idea.

>> During the compilation of applications, the preprocessor macro
>> Py_LIMITED_API must be defined. Doing so will hide all definitions
>> that are not part of the ABI.
> 
> So extensions wanting to use the full Python C-API as documented
> in the C-API docs will still be able to do this, right ?

Yep - they just wouldn't define the new macro.

>> Type Objects
>> ------------
>>
>> The structure of type objects is not available to applications;
>> declaration of "static" type objects is not possible anymore
>> (for applications using this ABI).
> 
> Hmm, that's going to create big problems for extensions that
> want to expose a C-API for their types: Type checks are normally
> done by pointer comparison using those static type objects.

They would just have to expose "MyExtensionPrefix_MyType_Check" and
"MyExtensionPrefix_MyType_CheckExact" functions the same way that types
in the C API do.

>> Functions and function-like Macros
>> ----------------------------------
>>
>> Function-like macros (in particular, field access macros) remain
>> available to applications, but get replaced by function calls
>> (unless their definition only refers to features of the ABI, such
>> as the various _Check macros)
> 
> Including Py_INCREF()/Py_DECREF() ?

I believe so - MvL deliberately left the fields that the ref counting
relies on as part of the ABI.

>> Excluded Functions
>> ------------------
>>
>> Functions declared in the following header files are not part
>> of the ABI:
>> - cellobject.h
>> - classobject.h
>> - code.h
>> - frameobject.h
>> - funcobject.h
>> - genobject.h
>> - pyarena.h
>> - pydebug.h
>> - symtable.h
>> - token.h
>> - traceback.h
> 
> I don't think that's feasable: you basically remove all introspection
> functions that way.
> 
> This will need a more fine-grained approach.

I don't think it is reasonable to expect the introspection interfaces to
remain stable at a binary level across versions.

Having "I want deep introspection support from C" and "I want to use a
single binary for multiple Python versions" be mutually exclusive
choices sounds like a perfectly sensible position to me.

Also, keep in mind that even an extension module that restricts itself
to Py_LIMITED_API would still be able to call in to the Python
equivalents via PyObject_Call and friends (e.g. by importing and using
the inspect and traceback modules).

> What if you mix extensions that use the full C-API with ones
> that restrict themselves to the limited version ?
> 
> Would creating a Python object in a full-API extension and
> free'ing it in a limited-API extension cause problems ?

Possibly, if you end up mixing C runtimes in the process. Specifically:
1. Python linked with MSVCRT X
2. Full extension module linked with MSVCRT Y
3. Limited extension module linked with MSVCRT Z

The PyMem/PyObject APIs in the limited extension module will use the
heap in MSVCRT X, since they will be redirected through the Python
stable ABI as function calls. However, if the full extension module uses
the macro forms and links with the wrong MSVCRT version, then you have
the usual opportunities for conflicts between the two C runtimes.

This isn't a problem created by defining a stable ABI though - it's the
main reason mixing C runtimes is a bad idea. (The two others we have
noted so far being IO issues, especially attempting to share FILE*
instances and the fact that changing the locale will only affect
whichever runtime the extension module linked against).

>> Good point. As a separate issue, I would actually like to deprecate,
>> then remove these APIs. I had originally hoped that this would happen
>> for 3.0 already, alas, nobody worked on it.
>>
>> In any case, I have removed them from the ABI now.
> 
> How do you expect Python extensions to allocate memory and objects
> in a platform independent way without those APIs ?
> 
> And as an aside: Which API families are you referring to ? PyMem_Malloc,
> PyObject_Malloc, or PyObject_New ?

The ones with a FILE* parameter in the signature. There's no problem
with the PyMem/PyObject functions since those will be redirected to
consistently use the version of the C runtime that Python was originally
linked against (their macro counterparts are obviously off limits for
the stable ABI).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------


More information about the Python-Dev mailing list