[Python-Dev] PEP 384: Defining a Stable ABI

[M.-A. Lemburg]

Martin v. Löwis wrote:
> Thomas Wouters reminded me of a long-standing idea; I finally
> found the time to write it down.
> 
> Please comment!
> ...
>

Up until this PEP proposal, we had a very simple scheme for
the Python C-API: all documented functions and variables with
a "Py" prefix were part of the C-API, everything else was not
and could change between releases (in particular the private
"_Py" prefix APIs).

Changing the published APIs was considered a bad thing in the
2.x development process and generally required a good reason
to get supported.

Changing private functions or ones that were not documented
was generally never a big problem.

Now, with the PEP, I have a feeling that the Python C-API
will in effect be limited to what's in the PEP's idea of
a usable ABI and open up the non-inluded public C-APIs
to the same rate of change as the private APIs.

If that's the case, the PEP should be discussed on the C-API
list first, in order to identify a complete list of APIs that
is supposed to define the Python C-API. Ideally, all other
APIs would then need to be made private. However, I doubt that
this is possible before switching to Python 4.0.

Then again, I'm not sure whether that's what you're aiming for...

An optional cross-version ABI would certainly be a good thing.

Limiting the Python C-API would be counterproductive.

> During the compilation of applications, the preprocessor macro
> Py_LIMITED_API must be defined. Doing so will hide all definitions
> that are not part of the ABI.

So extensions wanting to use the full Python C-API as documented
in the C-API docs will still be able to do this, right ?

> Type Objects
> ------------
> 
> The structure of type objects is not available to applications;
> declaration of "static" type objects is not possible anymore
> (for applications using this ABI).

Hmm, that's going to create big problems for extensions that
want to expose a C-API for their types: Type checks are normally
done by pointer comparison using those static type objects.

> Functions and function-like Macros
> ----------------------------------
> 
> Function-like macros (in particular, field access macros) remain
> available to applications, but get replaced by function calls
> (unless their definition only refers to features of the ABI, such
> as the various _Check macros)

Including Py_INCREF()/Py_DECREF() ?

> Excluded Functions
> ------------------
> 
> Functions declared in the following header files are not part
> of the ABI:
> - cellobject.h
> - classobject.h
> - code.h
> - frameobject.h
> - funcobject.h
> - genobject.h
> - pyarena.h
> - pydebug.h
> - symtable.h
> - token.h
> - traceback.h

I don't think that's feasable: you basically remove all introspection
functions that way.

This will need a more fine-grained approach.

> Linkage
> -------
> 
> On Windows, applications shall link with python3.dll;

You mean: extensions that were compiled with Py_LIMITED_API, right ?

> an import
> library python3.lib will be available. This DLL will redirect all of
> its API functions through /export linker options to the full
> interpreter DLL, i.e. python3y.dll.

What if you mix extensions that use the full C-API with ones
that restrict themselves to the limited version ?

Would creating a Python object in a full-API extension and
free'ing it in a limited-API extension cause problems ?

> Implementation Strategy
> =======================
> 
> This PEP will be implemented in a branch, allowing users to check
> whether their modules conform to the ABI. To simplify this testing, an
> additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
> type object layout, to let users postpone rewriting all types. When
> the this branch is merged into the 3.2 code base, this macro will
> be removed.

Now I'm confused again: this sounds a lot like you do want all extension
writers to only use the limited API.

[And in another post]
>> Something I haven't seen explicitly mentioned as yet (in the PEP or the
>> > python-dev list discussion) are the memory management APIs and the FILE*
>> > APIs which can cause the MSVCRT versioning issues on Windows.
>> > 
>> > Those would either need to be excluded from the stable ABI or else
>> > changed to use opaque pointers.
> 
> Good point. As a separate issue, I would actually like to deprecate,
> then remove these APIs. I had originally hoped that this would happen
> for 3.0 already, alas, nobody worked on it.
> 
> In any case, I have removed them from the ABI now.

How do you expect Python extensions to allocate memory and objects
in a platform independent way without those APIs ?

And as an aside: Which API families are you referring to ? PyMem_Malloc,
PyObject_Malloc, or PyObject_New ?

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 25 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2009-06-29: EuroPython 2009, Birmingham, UK                34 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/