Hi,
Last months, I was busy to fill https://pythoncapi.readthedocs.io/
website with random notes. Many discussions occurred on this list and
python-dev, but I was only able to make the most simple and least
controversal changes in Python upstream. I didn't write a PEP because
CPython had a governance crisis. Since a new Steering Committee has
been elected, it's time to see how concrete PEP can be written.
IMHO we need to split the giant "C API" problem into multiple PEPs. I
propose 4 PEPs:
PEP A: Ecosystem of C extensions in 2019
PEP B: Define good and bad C APIs
PEP C: Plan to enhance the existing C API
PEP D: Completely new C API
There is also an ongoing discussion about embedded Python and Python
initialization API, but I'm scared by this topic so I don't even
propose to write a new PEP which would supersed PEP 432 :-)
https://bugs.python.org/issue22213
== PEP A: Ecosystem of C extensions in 2019 ==
Discuss cffi, Cython, PyQt usage of the stable ABI, CPython C API,
etc. The goal is not to solve any problem, mostly to list existing
options.
It sounds like an unsual PEP, but I think that a PEP is needed since
the same discussions happened multiple times.
This PEP can describe what are the kind of "C extensions" and maybe
suggest which tools are the best depending on the kind. cffi doesn't
cover all cases, the C API isn't always the right answer, etc.
== PEP B: Define good and bad C APIs ==
https://pythoncapi.readthedocs.io/bad_api.html can be used as a
starting point. It should be an informal PEP which evolves as PEP 7
and PEP 8 are evolving.
== PEP C: Plan to enhance the existing C API ==
This one sounds like the most controversial PEP :-) I see different things:
* Plan to deprecate and remove bad APIs
* Plan to help C extensions maintainers to move away from these bad APIs
* Plan to test the stability of the API
* Plan to test the stability of the ABI
The even more controversial idea: provide multiple Python runtimes for
CPython, not only one:
https://pythoncapi.readthedocs.io/runtimes.html
== PEP D: Completely new C API ==
Well, that's the obvious alternative to PEP C.
Armin Rigo's PyHandle idea may be a good start?
https://pythoncapi.readthedocs.io/pyhandle.html
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
It seems to me that all the new proposals, whether just hiding existing
implementation details or moving to some new object id or tagged
pointer system, will require extension module writers to replace
PyObject *
with
PyHandle
And extension writers should also not be accessing struct fields in
objects or type records directly, so we also want to replace eg
type->tp_str
with some kind of macro or inline function.
So as a first step, we should add a typedef for PyHandle to Python.h
(well, most likely within object.h) and macros defined for all or the most
common object / type struct members.
We can do this now without breaking any existing code. (Uh, right?)
Then interested people can experiment with new ideas and APIs and
we all get some idea of how hard a final changeover would be.
My other suggestion is that once the new typedefs and macros are
in Python.h, announce that Python 4.0 won't guarantee source code
compatibility for existing extension code.
--
cheers,
Hugh Fisher
[Adding back the list - I assume dropping it was an accidenty]
On 04Mar2019 1234, Neil Schemenauer wrote:
> On 2019-03-03, Steve Dower wrote:
>> In my opinion, you can dislike many of the Windows-specific
>> "enhancements" around COM (like DCOM, etc., and I do dislike them), but
>> the core concepts are very well proven, including being used from
>> JavaScript and .NET (fully GC languages). Perhaps moreso than JNI?
>
> Thanks Steve. You are correct, we should learn from COM too. Do
> you have any suggested references? I poked around some in the
> CoreCLR repo on github. E.g. this was pretty interesting:
>
> https://github.com/dotnet/coreclr/blob/master/Documentation/botr/exceptio...
>
> Are there documents in the open source "dotnet" that would be
> relevant to the COM design implementation? I think a challenge with
> COM is that it is more comprehensive and therefore more complicated.
> So, for outsiders, it could be more difficult to understand how it
> works.
I'd suggest looking at the design notes here instead:
https://github.com/Microsoft/xlang
This is the cross-platform implementation of the "core" of COM
(basically, the cross-language ABI part without necessarily including
the magic cross-process/machine and proxy/marshalling support that is
considered part of COM on Windows).
There is also a tool in that repo for generating Python C extensions to
project objects defined in xlang/COM into Python. Since most of the new
Windows API (from Win8 onwards) is defined like this, that's their first
examples, but it's not at all tied to Windows.
Cheers,
Steve
Hi,
has anyone considered mixing the "opaque handle" idea with tagged pointers?
I could imagine (on 64 bits) to reserve, say, a 'signed' 24 bits for a
refcount and the rest for an index into an array of object/vtable pointer
structs. All negative refcounts would have special meanings, such as
- this is the immortal None
- this is actually a tagged integer and not an object index
- this is a tagged float with an exact 32bit (or integer) value
- refcount has overflown and object has become immortal :)
Things like these, can't say which are reasonable and/or fast enough. I
could also imagine reserving a few more bits for a builtin type ID to speed
up type checks, especially for int/float/list/tuple/dict/fast-callable,
maybe also things like flat tuples, where the items are directly stored
consecutively in the object array following the tuple index.
Other runtimes could then implement the handles differently, as really
opaque values and with or without tagged pointers, but CPython could use
its own macros to give meaning to the tags. 32bit architectures would
probably require different macros also for CPython, but since the whole
design would need to work with and without tags, that should be doable.
Does this seem like something worth discussing?
Stefan
On 2019-02-28, Carl Shapiro wrote:
> Because of all of the accumulated experience with handles in other systems,
> I think CPython is positioned to do much better than its predecessors.
I spent some time last night reading about JNI and I see that it
solves many of the problems we are trying to solve. Certainly we
should learn from it.
You can download a PDF copy of the JNI book here:
http://java.sun.com/docs/books/jni/
This looks like a useful article as well, outlining common mistakes
when using the JNI:
https://www.ibm.com/developerworks/library/j-jni/index.html
The JNI book is pretty old so I'm not sure if JNI has evolved a lot
since then. However, after only skimming the book last night, I
find lots of interesting ideas.
First, JNI passes a JNIEnv pointer as the first argument of all
native methods. I think it is similar to our threadstate structure.
Explicitly passing it avoids some problems. Since Java doesn't have
a GIL, smoothly handling threading is a big deal. I don't know if
we should emulate that and pass threadstate (or something similar)
as well.
At a recent core sprint, I recall discussing an idea like that with
Dino and Carl. E.g. a new flag for extension modules that would
make CPython pass the threadstate to extension functions. I'm
pretty ignorant when it comes to multi-threading but I think those
guys thought looking it up in thread local storage might be quick
enough, rather explicitly passing it everywhere.
Rather than the JNI API being functions you can call, like
PyObject_Something(x), they are implemented as a vtable on the
JNIEnv structure. So, you do something like:
Java_do_something(JNIEnv *env, jobject obj)
{
(*env)->DoSomething(env, obj)
}
JNI provides strict binary compatiblity so you really can't have
macros or inlined functions as part of the API. This vtable idea
has some nice advantages. You can start the JVM with different
command line parameters and a different vtable can be used. CPython
does something like this for tracemalloc. The JNI way seems
cleaner and maybe more powerful.
Using a macro would seem cleaner to me, e.g.
#define DoSomething(env, obj) ((*env)->DoSomething(env, ob))
Java_do_something(JNIEnv *env, jobject obj)
{
DoSomething(env, obj);
}
Maybe we could have it both ways (binary compatiblity or lower
overhead). Use an inline function like the following:
static inline void
DoSomething(JNIEnv *env, jobject obj)
{
#ifdef STABLE_BINARY_INTERFACE
((*env)->DoSomething(env, ob))
#else
... inline implementation of env->DoSomething
#fi
}
There are three kinds of opaque object references (handles): local
references, global references and weak global references. As I
understand, local references are a handle that gets closed when your
native method returns. That sounds useful and makes life easier
for extension authors (harder to leak memory if forgetting to close
handles). You are limited in the number of local references you can
use (default 16?) but the limit can be increased. You can also
explicitly close local handles so you don't run out or so you free
large chunks of memory. E.g.
lref = ... /* a large Java object */
...
(*env)->DeleteLocalRef(env, lref);
Local references sound very much like what Carl Shapiro and Larry
Hastings were suggesting as a way to deal with borrowed references
in the CPython API. I.e. make them a local reference and then close
them when native function returns.
Global references are what I was thinking of for the PyHandle API.
They would live beyond your native function call and you have to
remember to close them. Weak global references are pretty
obviously. We would want to provide them too.
JNI uses a similar scheme to CPython to deal with errors. I.e. JNI
methods typically return NULL on error and set something inside the
JNIEnv structure to record the details of the error. They have a
method that is like PyErr_Occurred(), e.g.
if ((*env)->ExceptionCheck(env)) {
return NULL // error case
}
They spell out explicitly which JNI methods are safe to call when an
error has occurred. In the JNI book, they say:
It is extremely important to check, handle, and clear a pending
exception before calling any subsequent JNI functions.
I gather this is a source of many bugs. I wonder if it would be
better to return an object that enforces correct error handling.
One example I found was the LLVM Error class:
https://llvm.org/doxygen/classllvm_1_1Error.html#details
I don't know how you would implement something like that in C.
Maybe returning NULL is okay as it is working for JNI and matches
what CPython does internally.
The JNI has to solve a similar problem to Python and provide a rich
set of accessor functions for object handles. The JNI approach
works no matter how the Java virtual machine represents objects
internally. This abstration has a cost and so they provide a faster
way for repeated access to primitive data types, such as arrays and
strings. E.g. a function that gets a "pinned" version of the array
elements.
JNI provides native access to fields and methods of Java objects.
The JNI identifies methods and fields by their symbolic names and
type descriptors. A two-step process factors out the cost of
locating the field or method from its name and descriptor. For
example, to read an integer instance field i in class cls, native
code first obtains a field ID, as follows:
jfieldID fid = env->GetFieldID(env, cls, "i", "I");
The native code can then use the field ID repeatedly, without the
cost of field lookup, as follows:
jint value = env->GetIntField(env, obj, fid);
There are rules about how the field ID can be cached. The advantage
of this design is that JNI does not impose any restrictions on how
field and method IDs are implemented internally.
Regards,
Neil
I've had enough ideas bouncing around in my head that I had to get them
written up :)
So I'm proposing to produce an informational PEP to describe what a
"good" C API looks like and act as guidance as we implement new APIs or
change existing ones.
This is a rough, incomplete first draft that nonetheless I think is
enough to trigger useful discussions. It's a brain dump, but I've
already dumped most of this before.
They're in the text below, but I'll repeat here:
* this is NOT a brand-new API
* this is NOT exactly what we currently have implemented
* this is NOT a proposal to stop shipping half the standard library
* this IS meant to provide context for discussing both the issues with
our current API and to help drive discussions of any new API or API changes
I don't have any particular desire to own the entire doc, so if anyone
wants to become a co-author I'm very open to that. However, I do have
strong opinions on this topic after a number of years working with
*excellent* API designs, designers and processes. If you want to propose
a _totally_ different vision from this, please consider writing an
alternative rather than trying to co-opt this one :)
(Doc in approximate Markdown, automatically wrapped to 72 cols for email
and I haven't checked if that broke stuff. Sorry if it did)
Cheers,
Steve
---
CPython C API Design Guidelines
# Abstract
This document is intended to be a set of guiding principles for
development of the current CPython C API. Future additions and
enhancements to the CPython C API should follow, or at least be
influenced by, the principles described here. At a minimum, any new or
modified C APIs should be able to be categorised according to the
terminology defined here, even if exceptions have to be made.
# Things this document is NOT
This document is NOT a design of a completely new API (though a
hypothetical new API should follow this design).
This document is NOT documentation of the current API (though the
current API should come to resemble it over time).
This document is NOT a set of binding rules in the same sense as PEP 7
and PEP 8 (though designs should be tested against it and exceptions
should be rare).
This document is NOT permission to make backwards-incompatible
modifications to the current API (though backwards-incompatible
modifications should still be made where warranted).
# Definitions
A common understanding of certain terms is necessary to talking about
the CPython C API. This section has two goals: to clarify existing
common terminology, and to introduce new terminology. Terms are
presented in a logical order, rather than alphabetically.
## Existing terms
**Application**: Any independent program that can be launched directly.
Compare and contrast with *extension*. CPython is normally considered an
application.
**Extension**: A program that integrates into an application, and cannot
be launched directly but must be loaded by that application. Python
modules, native or otherwise, are considered extenions. When embedded
into another application, CPython is considered an extension.
**Native extension**: A subset of all extensions that are compiled to
the same language as the application they integrate with. When embedded
into an application that is written in C or uses C-compatible
conventions, CPython is considered a native extension.
**API**: Application Programming Interface. The set of interactions
defined by an application to allow extensions to extend, control, and
interact with the first. Typically refers to OOP objects and functions
in the abstract. CPython has one API that applies for all scenarios in
all contexts, though each scenario will likely only use a subset of this
API.
**ABI**: Application Binary Interface. The implementation of an API such
that its interactions can be realized by a digital computer. Typically
includes memory layouts and binary representations, and is a function of
the build tools used to compile CPython. CPython has different ABIs in
different contexts, and a different ABI for native extensions compared
to extensions.
**Stdlib**: Standard library. Components that build upon the Python
language in order to provide useful building blocks and pre-written
functionality for users.
## New terms
These terms are introduced briefly here and described in much greater
detail below.
**API ring**: One subset of an API for the purpose of extension
compatibility. Extensions to CPython care about rings. Extensions choose
to target a particular ring to trade off between deeper integration and
tighter coupling. Targeting one ring includes access to all rings
outside of that one. Rings are orthogonal to layers.
**API layer**: One subset of an API for the purpose of application and
internal compatibility. Applications that embed CPython, and the CPython
implementation itself, cares about layers. Applications choose to adopt
or implement a particular layer, implicitly including all lower layers.
Layers are orthogonal to rings.
# Quick Overview
For context as you continue reading, these are the API **rings**
provided by CPython:
* Python ring (equivalent of the Python language)
* CPython ring (CPython-specific APIs)
* Internal ring (intended for internal use only)
These are the API **layers** provided by CPython:
* Optional stdlib layer (dependencies that must be explicitly required)
* Required stdlib layer (dependencies that can be assumed)
* Platform adaption layer (ability to interact with the platform)
* Core layer ("pure" mode with no platform interactivity)
(Reminder that this document does not reflect the current state of
CPython, but is both aspirational and defining terms for the purposes of
discussion. This is not a proposal to remove anything from the standard
distribution!)
# API Rings
CPython provides three API rings, listed here from outermost to
innermost:
* Python ring
* CPython ring
* Internal ring
An extension that targets the Python ring does not have access to the
CPython or Internal rings. Likewise, an extension that targets the
CPython ring does not have access to the Internal ring, but does use the
Python ring.
When CPython is an extension of another application, that application
can also select which ring to target.
The expectation is that all Python implementations can provide an
equivalent Python ring, CPython officially supports extensions using the
CPython ring when targeting CPython, and the Internal ring is available
but unsupported.
## Python API ring
The Python ring provides functionality that should be equivalent across
all Python implementations - in essence, the Python language itself
defines this ring.
The C implementation of the Python API allows native code to interact
with Python objects as if it were written in Python. The Python API
supports duck-typing and should correctly handle the substitution of
alternative types.
For a concrete example, `PyObject_GetItem` is part of the Python ring
while `PyDict_GetItem` is in the CPython ring.
Compatibility requirements for the Python API match the language
version. Specifically, code relying on the Python API should only break
or change behaviour if the equivalent code written in Python would also
break or change behaviour.
For CPython, including `Python.h` should only provide access to the
Python ring. Accessing any other rings should produce a compile error.
## CPython API ring
The CPython ring provides functionality that is specific to CPython.
Extensions that opt in to the CPython ring are tied directly to CPython,
but have access to functions that are specific to CPython.
Functions in the CPython ring may require the caller to be using C or be
able to provide C structures allocated in memory.
In general, most applications that embed CPython will use the CPython
ring. Also, native extensions in the Optional stdlib layer
For a concrete example, the `PyCapsule` type belongs in the CPython ring
(that is, other implementations are not required to provide this
particular way to smuggle C pointers through Python objects).
As a second concrete example, `PyType_FromSpec` belongs in the CPython
ring. (The equivalent in the Python ring would be to call the `type`
object, while the equivalent in the internal ring would be to define a
statis `PyTypeObject`.)
Compatibility requirements for the CPython API match the CPython
major.minor version. Specifically, code relying on the CPython API
should only break or change behaviour if the major.minor version
changes.
For CPython, as well as `Python.h`, also include `cpython/<header>.h` to
obtain access to APIs in the CPython ring.
## Internal API ring
The Internal ring provides functionality that is used to implement
CPython. Extensions that opt in to the Internal ring may need to rebuild
for every CPython build.
In general, most of the Required stdlib layer will use the Internal
ring.
For CPython, as well as `Python.h`, also include `internal/<header>.h`
to obtain access to APIs in the Internal ring.
# API Layers
CPython provides four API layers, listed here from top to bottom:
* Optional stdlib layer
* Required stdlib layer
* Platform adaptation layer
* Core layer
An application embedding Python targets one layer and all those below
it, which affects the functionality available in Python.
Higher layers may depend on the APIs provided by lower layers, but not
the other way around. In general, layers should aim to maximise
interaction with the next layer down and avoid skipping it, but this is
not a strict requirement.
Lower layers are required to maintain backwards compatibility more
strictly than the layers above them.
Components within a layer that depend on other components within that
layer must be treated as a single component for determining whether it
may be included or omitted.
Standard Python distributions (that is, anything that may be launched
with the `python` command) will depend upon most components in the
Optional stdlib layer, and hence will require _everything_ from the
Required stdlib layer and below. Only embedders and potentially
deployment tools will use reduced layers.
(Reminder: this document does not present the current state of CPython.)
## Core layer
This layer is the core language and evaluation engine. By adopting this
layer, an application can provide platform-independent Python execution.
However, it may require providing implementations of a number of
callbacks in order to be functional (e.g. for dynamic memory
allocation).
Examples of current components that fit into the core layer:
* Most of most built-in types (str, int, list, dict, etc.)
* compile, exec, eval
* read-only members of the sys module
* import
Important but potentially non-obvious implications of relying only on
the core layer:
* Dynamic memory allocation/deallocation is part of the Platform
adaptation layer, but there is no way to avoid it here. So any user of
the core API will need to provide allocators and deallocators. The
CPython Platform adaptation layer provides the "default"
implementations, but if an embedder does not want to use these then
targeting the Core layer will omit them.
* File system and standard streams are part of the Platform adaptation
layer, which leaves `open` and `sys.stdout` (among others) without a
default implementation. An application that wants to support these
without adding more layers needs to provide its own implementations
* The core layer only exposes UTF-8 APIs. Encoding and decoding for the
current platform requires the Platform adaptation layer, while arbitrary
encoding and decoding requires the Optional stdlib layer.
* Imports in the core layer are satisfied by a "blind" callback. The
Platform adaptation layer provides the support for frozen, bytecode and
natively-encoded source imports, while the Optional stdlib layer is
required for arbitrary encodings in source files
## Platform adaptation layer
This layer provides the CPython implementation of platform-specific
adapters to support the core layer.
* Memory allocation/deallocation
* File system access
* Standard input/output streams
* Cryptographic random number generation
* os module
* CPython imports
Important but potentially non-obvious implications of relying only on
the platform adaptation layer:
* File system access generally requires text encodings, but the full set
of codecs are in the optional stdlib layer. To fully separate these
layers, an implementation of the current file system encoding would be
required in the Platform adaptation layer. (But arbitrarily
encoding/decoding the _contents_ of a file may require higher layers.)
* Importing from source code may also require arbitrary encodings, but
imports that can be fully satisfied without this are provided here (e.g.
native extension modules, precompiled bytecode, frozen modules, natively
encoded source files)
## Required stdlib layer
This layer provides common APIs for interactions between other modules.
All components in the Optional stdilib layer may assume that if _they_
are present, everything in this layer is also present.
* standard ABCs
* compiler services (e.g. `copy`, `functools`, `traceback`)
* standard interop types (e.g. `pathlib`, `enum`, `dataclasses`)
## Optional stdlib layer
This layer provides modules that fundamentally stand alone. None of the
lower levels may depend on these components being present, and
components in this layer should explicitly declare dependencies on
others in the same layer.
This layer is valuable for embedders and distributors that want to omit
certain functionality. For example, omitting `socket` should be possible
when that functionality is not required, as it is in the Optional stdlib
layer, and omitting it should only affect those components in the
Optional stdlib layer that have explicitly required it.
* platform-independent algorithms (e.g. `itertools`, `statistics`)
* application-specific functionality (e.g. `email`, `socket`, `ftplib`,
`ssl`)
* additional compiler services (e.g. `ast`)
* text codecs (e.g. `base64`, `codecs`, `encodings`)
* Python-level FFI (e.g. `ctypes`)
* tools (e.g. ``idlelib``, ``pynche``, ``distutils``, ``msilib``)
* configuration/information (e.g. ``site``, ``sysconfig``, ``platform``)
Components in the Optional stdlib layer may be independently versioned.
