[Python-Dev] [unladen-swallow] PEP 384: Defining a Stable ABI

[Jeffrey Yasskin]

A couple thoughts:

I'm with the people who think the refcount should be accessed through
functions by apps that want ABI compatibility. In particular,
GIL-removal efforts are guaranteed to change how the refcount is
modified, but there's a good chance they wouldn't have to change the
API. (We have some ideas for how to maintain source compatibility in
the absence of a GIL:
http://code.google.com/p/unladen-swallow/wiki/ExtensionModules#Reference_Counting)
Over an 8-year lifetime for Python 3, Moore's law predicts that
desktop systems will have up to 64 cores, at which point even the
simplest GIL-removal strategy of making refcounts atomic will be a
win, despite the 2x performance loss for a single thread. I wouldn't
want an ABI to rule that out.

I do think the refcounting macros should remain present in the API
(not ABI) for apps that only need source compatibility and want the
extra speed.

I wonder if it makes sense to specify an API compatibility mode in
this PEP too.

"Py_LIMITED_API" may not be the right macro name—it didn't imply
anything about an ABI when I first saw it. Might it make sense to use
Py_ABI_COMPATIBILITY=### instead? (Where ### could be an ISO date like
20090520.) That would put "ABI" in the macro name and make it easier
to define new versions later if necessary. (New versions would help
people compile against a new version of Python and be confident they
had something that would run against old versions.) If we never define
a new version, defining it to a number instead of just anything
doesn't really hurt.

It's probably worth pointing out in the PEP that the fact that
PyVarObject.ob_size is part of the ABI means that PyObject cannot
change size, even by adding fields at the end.

Right now, the globals representing types are defined like
"PyAPI_DATA(PyTypeObject) PyList_Type;". To allow the core to use the
new type creation functions, it might be useful to make the ABI type
objects PyTypeObject* constants instead.

In general, this looks really good. Thanks!

Jeffrey

On Sun, May 17, 2009 at 1:54 PM, "Martin v. Löwis" <martin at v.loewis.de> wrote:
>
> Thomas Wouters reminded me of a long-standing idea; I finally
> found the time to write it down.
>
> Please comment!
>
> Regards,
> Martin
>
> PEP: 384
> Title: Defining a Stable ABI
> Version: $Revision: 72754 $
> Last-Modified: $Date: 2009-05-17 21:14:52 +0200 (So, 17. Mai 2009) $
> Author: Martin v. Löwis <martin at v.loewis.de>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 17-May-2009
> Python-Version: 3.2
> Post-History:
>
> Abstract
> ========
>
> Currently, each feature release introduces a new name for the
> Python DLL on Windows, and may cause incompatibilities for extension
> modules on Unix. This PEP proposes to define a stable set of API
> functions which are guaranteed to be available for the lifetime
> of Python 3, and which will also remain binary-compatible across
> versions. Extension modules and applications embedding Python
> can work with different feature releases as long as they restrict
> themselves to this stable ABI.
>
> Rationale
> =========
>
> The primary source of ABI incompatibility are changes to the lay-out
> of in-memory structures. For example, the way in which string interning
> works, or the data type used to represent the size of an object, have
> changed during the life of Python 2.x. As a consequence, extension
> modules making direct access to fields of strings, lists, or tuples,
> would break if their code is loaded into a newer version of the
> interpreter without recompilation: offsets of other fields may have
> changed, making the extension modules access the wrong data.
>
> In some cases, the incompatibilities only affect internal objects of
> the interpreter, such as frame or code objects. For example, the way
> line numbers are represented has changed in the 2.x lifetime, as has
> the way in which local variables are stored (due to the introduction
> of closures). Even though most applications probably never used these
> objects, changing them had required to change the PYTHON_API_VERSION.
>
> On Linux, changes to the ABI are often not much of a problem: the
> system will provide a default Python installation, and many extension
> modules are already provided pre-compiled for that version. If additional
> modules are needed, or additional Python versions, users can typically
> compile them themselves on the system, resulting in modules that use
> the right ABI.
>
> On Windows, multiple simultaneous installations of different Python
> versions are common, and extension modules are compiled by their
> authors, not by end users. To reduce the risk of ABI incompatibilities,
> Python currently introduces a new DLL name pythonXY.dll for each
> feature release, whether or not ABI incompatibilities actually exist.
>
> With this PEP, it will be possible to reduce the dependency of binary
> extension modules on a specific Python feature release, and applications
> embedding Python can be made work with different releases.
>
> Specification
> =============
>
> The ABI specification falls into two parts: an API specification,
> specifying what function (groups) are available for use with the
> ABI, and a linkage specification specifying what libraries to link
> with. The actual ABI (layout of structures in memory, function
> calling conventions) is not specified, but implied by the
> compiler. As a recommendation, a specific ABI is recommended for
> selected platforms.
>
> During evolution of Python, new ABI functions will be added.
> Applications using them will then have a requirement on a minimum
> version of Python; this PEP provides no mechanism for such
> applications to fall back when the Python library is too old.
>
> Terminology
> -----------
>
> Applications and extension modules that want to use this ABI
> are collectively referred to as "applications" from here on.
>
> Header Files and Preprocessor Definitions
> -----------------------------------------
>
> Applications shall only include the header file Python.h (before
> including any system headers), or, optionally, include pyconfig.h, and
> then Python.h.
>
> During the compilation of applications, the preprocessor macro
> Py_LIMITED_API must be defined. Doing so will hide all definitions
> that are not part of the ABI.
>
> Structures
> ----------
>
> Only the following structures and structure fields are accessible to
> applications:
>
> - PyObject (ob_refcnt, ob_type)
> - PyVarObject (ob_base, ob_size)
> - Py_buffer (buf, obj, len, itemsize, readonly, ndim, shape,
>  strides, suboffsets, smalltable, internal)
> - PyMethodDef (ml_name, ml_meth, ml_flags, ml_doc)
> - PyMemberDef (name, type, offset, flags, doc)
> - PyGetSetDef (name, get, set, doc, closure)
>
> The accessor macros to these fields (Py_REFCNT, Py_TYPE, Py_SIZE)
> are also available to applications.
>
> The following types are available, but opaque (i.e. incomplete):
>
> - PyThreadState
> - PyInterpreterState
>
> Type Objects
> ------------
>
> The structure of type objects is not available to applications;
> declaration of "static" type objects is not possible anymore
> (for applications using this ABI).
> Instead, type objects get created dynamically. To allow an
> easy creation of types (in particular, to be able to fill out
> function pointers easily), the following structures and functions
> are available::
>
>  typedef struct{
>    int slot;    /* slot id, see below */
>    void *pfunc; /* function pointer */
>  } PyType_Slot;
>
>  struct{
>    const char* name;
>    const char* doc;
>    int basicsize;
>    int itemsize;
>    int flags;
>    PyType_Slot *slots; /* terminated by slot==0. */
>  } PyType_Spec;
>
>  PyObject* PyType_FromSpec(PyType_Spec*);
>
> To specify a slot, a unique slot id must be provided. New Python
> versions may introduce new slot ids, but slot ids will never be
> recycled. Slots may get deprecated, but continue to be supported
> throughout Python 3.x.
>
> The slot ids are named like the field names of the structures that
> hold the pointers in Python 3.1, with an added ``Py_`` prefix (i.e.
> Py_tp_dealloc instead of just tp_dealloc):
>
> - tp_dealloc, tp_print, tp_getattr, tp_setattr, tp_repr,
>  tp_hash, tp_call, tp_str, tp_getattro, tp_setattro,
>  tp_doc, tp_traverse, tp_clear, tp_richcompare, tp_iter,
>  tp_iternext, tp_methods, tp_base, tp_descr_set, tp_descr_set,
>  tp_init, tp_alloc, tp_new, tp_is_gc, tp_bases, tp_del
> - nb_add nb_subtract nb_multiply nb_remainder nb_divmod nb_power
>  nb_negative nb_positive nb_absolute nb_bool nb_invert nb_lshift
>  nb_rshift nb_and nb_xor nb_or nb_int nb_float nb_inplace_add
>  nb_inplace_subtract nb_inplace_multiply nb_inplace_remainder
>  nb_inplace_power nb_inplace_lshift nb_inplace_rshift nb_inplace_and
>  nb_inplace_xor nb_inplace_or nb_floor_divide nb_true_divide
>  nb_inplace_floor_divide nb_inplace_true_divide nb_index
> - sq_length sq_concat sq_repeat sq_item sq_ass_item was_sq_ass_slice
>  sq_contains sq_inplace_concat sq_inplace_repeat
> - mp_length mp_subscript mp_ass_subscript
> - bf_getbuffer bf_releasebuffer
>
> XXX Not supported yet: tp_weaklistoffset, tp_dictoffset
>
> The following fields cannot be set during type definition:
> - tp_dict tp_mro tp_cache tp_subclasses tp_weaklist
>
> Functions and function-like Macros
> ----------------------------------
>
> All functions starting with _Py are not available to applications.
> Also, all functions that expect parameter types that are unavailable
> to applications are excluded from the ABI, such as PyAST_FromNode
> (which expects a ``node*``).
>
> All other functions are available, unless excluded below.
>
> Function-like macros (in particular, field access macros) remain
> available to applications, but get replaced by function calls
> (unless their definition only refers to features of the ABI, such
> as the various _Check macros)
>
> ABI function declarations will not change their parameters or return
> types. If a change to the signature becomes necessary, a new function
> will be introduced. If the new function is source-compatible (e.g. if
> just the return type changes), an alias macro may get added to
> redirect calls to the new function when the applications is
> recompiled.
>
> If continued provision of the old function is not possible, it may get
> deprecated, then removed, in accordance with PEP 7, causing
> applications that use that function to break.
>
> Excluded Functions
> ------------------
>
> Functions declared in the following header files are not part
> of the ABI:
> - cellobject.h
> - classobject.h
> - code.h
> - frameobject.h
> - funcobject.h
> - genobject.h
> - pyarena.h
> - pydebug.h
> - symtable.h
> - token.h
> - traceback.h
>
> Global Variables
> ----------------
>
> Global variables representing types and exceptions are available
> to applications.
> XXX provide a complete list.
>
> XXX should restrict list of globals to truly "builtin" stuff,
> excluding everything that can also be looked up through imports.
>
> XXX may specify access to predefined types and exceptions through
> the interpreter state, with appropriate Get macros.
>
> Other Macros
> ------------
>
> All macros defining symbolic constants are available to applications;
> the numeric values will not change.
>
> In addition, the following macros are available:
>
> - Py_BEGIN_ALLOW_THREADS, Py_BLOCK_THREADS, Py_UNBLOCK_THREADS,
>  Py_END_ALLOW_THREADS
>
> Linkage
> -------
>
> On Windows, applications shall link with python3.dll; an import
> library python3.lib will be available. This DLL will redirect all of
> its API functions through /export linker options to the full
> interpreter DLL, i.e. python3y.dll.
>
> XXX is it possible to redirect global variables in the same way?
> If not, python3.dll would have to copy them, and we should verify
> that all available global variables are read-only.
>
> On Unix systems, the ABI is typically provided by the python
> executable itself. PyModule_Create is changed to pass ``3`` as the API
> version if the extension module was compiled with Py_LIMITED_API; the
> version check for the API version will accept either 3 or the current
> PYTHON_API_VERSION as conforming. If Python is compiled as a shared
> library, it is installed as both libpython3.so, and libpython3.y.so;
> applications conforming to this PEP should then link to the former.
>
> XXX is it possible to make the soname libpython.so.3, and still
> have some applications link to libpython3.y.so?
>
> Implementation Strategy
> =======================
>
> This PEP will be implemented in a branch, allowing users to check
> whether their modules conform to the ABI. To simplify this testing, an
> additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
> type object layout, to let users postpone rewriting all types. When
> the this branch is merged into the 3.2 code base, this macro will
> be removed.
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>