[Python-Dev] PEP 384: Defining a Stable ABI

["Martin v. Löwis"]

> Now, with the PEP, I have a feeling that the Python C-API
> will in effect be limited to what's in the PEP's idea of
> a usable ABI and open up the non-inluded public C-APIs
> to the same rate of change as the private APIs.

That's certainly not the plan. Instead, the plan is to have
a stable ABI. The policy on the API isn't affected, except
for restricting changes to the API that would break the ABI.

>> During the compilation of applications, the preprocessor macro
>> Py_LIMITED_API must be defined. Doing so will hide all definitions
>> that are not part of the ABI.
> 
> So extensions wanting to use the full Python C-API as documented
> in the C-API docs will still be able to do this, right ?

Correct. They would link to the version-specific DLL on Windows.

>> The structure of type objects is not available to applications;
>> declaration of "static" type objects is not possible anymore
>> (for applications using this ABI).
> 
> Hmm, that's going to create big problems for extensions that
> want to expose a C-API for their types: Type checks are normally
> done by pointer comparison using those static type objects.

I don't see the problem. During module initialization, you
create the type object and store it in a global variable, and
then both clients and the module compare against the stored
pointer.

>> Function-like macros (in particular, field access macros) remain
>> available to applications, but get replaced by function calls
>> (unless their definition only refers to features of the ABI, such
>> as the various _Check macros)
> 
> Including Py_INCREF()/Py_DECREF() ?

Yes, although some people are requesting that these become functions.

>> Excluded Functions
>> ------------------
>>
>> Functions declared in the following header files are not part
>> of the ABI:
>> - cellobject.h
>> - classobject.h
>> - code.h
>> - frameobject.h
>> - funcobject.h
>> - genobject.h
>> - pyarena.h
>> - pydebug.h
>> - symtable.h
>> - token.h
>> - traceback.h
> 
> I don't think that's feasable: you basically remove all introspection
> functions that way.
> 
> This will need a more fine-grained approach.

What specifically is it that you want to do in a module that you
couldn't do anymore?

>> On Windows, applications shall link with python3.dll;
> 
> You mean: extensions that were compiled with Py_LIMITED_API, right ?

Correct, see "Terminology" in the PEP.

> 
>> an import
>> library python3.lib will be available. This DLL will redirect all of
>> its API functions through /export linker options to the full
>> interpreter DLL, i.e. python3y.dll.
> 
> What if you mix extensions that use the full C-API with ones
> that restrict themselves to the limited version ?

Some link against python3.dll, others against python32.dll (say).

> Would creating a Python object in a full-API extension and
> free'ing it in a limited-API extension cause problems ?

No problem that I can see.

>> This PEP will be implemented in a branch, allowing users to check
>> whether their modules conform to the ABI. To simplify this testing, an
>> additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
>> type object layout, to let users postpone rewriting all types. When
>> the this branch is merged into the 3.2 code base, this macro will
>> be removed.
> 
> Now I'm confused again: this sounds a lot like you do want all extension
> writers to only use the limited API.

I certainly want to support as many modules as reasonable with the PEP.
Whether or not developers then chose to build version-independent
binaries is certainly outside the scope of the PEP - it only specifies
action items for Python, not for application authors.

>>> Something I haven't seen explicitly mentioned as yet (in the PEP or the
>>>> python-dev list discussion) are the memory management APIs and the FILE*
>>>> APIs which can cause the MSVCRT versioning issues on Windows.
>>>>
>>>> Those would either need to be excluded from the stable ABI or else
>>>> changed to use opaque pointers.
>> Good point. As a separate issue, I would actually like to deprecate,
>> then remove these APIs. I had originally hoped that this would happen
>> for 3.0 already, alas, nobody worked on it.
>>
>> In any case, I have removed them from the ABI now.
> 
> How do you expect Python extensions to allocate memory and objects
> in a platform independent way without those APIs ?

I have only removed functions from the ABI that have FILE* in their
signatures.

> And as an aside: Which API families are you referring to ? PyMem_Malloc,
> PyObject_Malloc, or PyObject_New ?

Neither. PyRun_AnyFileFlags and friends.

Regards,
Martin