[Python-Dev] PEP 384: Defining a Stable ABI

Tue May 26 12:42:37 EDT 2009

Martin v. Löwis wrote:
>> Now, with the PEP, I have a feeling that the Python C-API
>> will in effect be limited to what's in the PEP's idea of
>> a usable ABI and open up the non-inluded public C-APIs
>> to the same rate of change as the private APIs.
> 
> That's certainly not the plan. Instead, the plan is to have
> a stable ABI. The policy on the API isn't affected, except
> for restricting changes to the API that would break the ABI.

Thanks for clarifying this.

>>> During the compilation of applications, the preprocessor macro
>>> Py_LIMITED_API must be defined. Doing so will hide all definitions
>>> that are not part of the ABI.
>> So extensions wanting to use the full Python C-API as documented
>> in the C-API docs will still be able to do this, right ?
> 
> Correct. They would link to the version-specific DLL on Windows.

Good.

>>> The structure of type objects is not available to applications;
>>> declaration of "static" type objects is not possible anymore
>>> (for applications using this ABI).
>> Hmm, that's going to create big problems for extensions that
>> want to expose a C-API for their types: Type checks are normally
>> done by pointer comparison using those static type objects.
> 
> I don't see the problem. During module initialization, you
> create the type object and store it in a global variable, and
> then both clients and the module compare against the stored
> pointer.

Ah, good point !

>>> Function-like macros (in particular, field access macros) remain
>>> available to applications, but get replaced by function calls
>>> (unless their definition only refers to features of the ABI, such
>>> as the various _Check macros)
>> Including Py_INCREF()/Py_DECREF() ?
> 
> Yes, although some people are requesting that these become functions.

I'd opt against that, simply because it creates a lot of overhead
due to the function call and issues with cache locality.

>>> Excluded Functions
>>> ------------------
>>>
>>> Functions declared in the following header files are not part
>>> of the ABI:
>>> - cellobject.h
>>> - classobject.h
>>> - code.h
>>> - frameobject.h
>>> - funcobject.h
>>> - genobject.h
>>> - pyarena.h
>>> - pydebug.h
>>> - symtable.h
>>> - token.h
>>> - traceback.h
>> I don't think that's feasable: you basically remove all introspection
>> functions that way.
>>
>> This will need a more fine-grained approach.
> 
> What specifically is it that you want to do in a module that you
> couldn't do anymore?

See my reply to Nick: some of the functions are needed even
if you don't want to do introspection, such as Py_FatalError()
or PyTraceBack_Print().

BTW: Given the headline, I take it that the various type checking
macros in these header will still be available, right ?

>>> On Windows, applications shall link with python3.dll;
>> You mean: extensions that were compiled with Py_LIMITED_API, right ?
> 
> Correct, see "Terminology" in the PEP.

Good, thanks.

>>> an import
>>> library python3.lib will be available. This DLL will redirect all of
>>> its API functions through /export linker options to the full
>>> interpreter DLL, i.e. python3y.dll.
>> What if you mix extensions that use the full C-API with ones
>> that restrict themselves to the limited version ?
> 
> Some link against python3.dll, others against python32.dll (say).
> 
>> Would creating a Python object in a full-API extension and
>> free'ing it in a limited-API extension cause problems ?
> 
> No problem that I can see.

Can we be sure that the MSVCRT used by python35.dll stays compatible
to the one used by say python32.dll ? What if the CRT memory
management changes between MSVCRT versions ?

Another aspect to consider:

How will this work in the light of having multiple copies of
Python installed on a Windows machine ?

They implementation section suggests that python3.dll would always
redirect to the python3x.dll for which it was installed, ie. if
I have Python 3.5 installed, but then need to run some app with
Python 3.2, the installed python3.dll would then point back to the
python32.dll.

Now, if I start a Python 3.5 application which uses a limited
API extension, this would try to load python32.dll into the
Python 3.5 process. AFAIK, that's not possible due to the
naming conflicts.

>>> This PEP will be implemented in a branch, allowing users to check
>>> whether their modules conform to the ABI. To simplify this testing, an
>>> additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
>>> type object layout, to let users postpone rewriting all types. When
>>> the this branch is merged into the 3.2 code base, this macro will
>>> be removed.
>> Now I'm confused again: this sounds a lot like you do want all extension
>> writers to only use the limited API.
> 
> I certainly want to support as many modules as reasonable with the PEP.
> Whether or not developers then chose to build version-independent
> binaries is certainly outside the scope of the PEP - it only specifies
> action items for Python, not for application authors.

Thanks for the clarification.

>>>> Something I haven't seen explicitly mentioned as yet (in the PEP or the
>>>>> python-dev list discussion) are the memory management APIs and the FILE*
>>>>> APIs which can cause the MSVCRT versioning issues on Windows.
>>>>>
>>>>> Those would either need to be excluded from the stable ABI or else
>>>>> changed to use opaque pointers.
>>> Good point. As a separate issue, I would actually like to deprecate,
>>> then remove these APIs. I had originally hoped that this would happen
>>> for 3.0 already, alas, nobody worked on it.
>>>
>>> In any case, I have removed them from the ABI now.
>> How do you expect Python extensions to allocate memory and objects
>> in a platform independent way without those APIs ?
> 
> I have only removed functions from the ABI that have FILE* in their
> signatures.
> 
>> And as an aside: Which API families are you referring to ? PyMem_Malloc,
>> PyObject_Malloc, or PyObject_New ?
> 
> Neither. PyRun_AnyFileFlags and friends.

Good.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 26 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2009-06-29: EuroPython 2009, Birmingham, UK                33 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/