On Sun, 18 Apr 2021, 2:47 am Antoine Pitrou, <antoine@python.org> wrote:
On Sun, 18 Apr 2021 02:13:57 +1000
Nick Coghlan <ncoghlan@gmail.com> wrote:
> If
> they want automatic resource management, then we want them working out how
> to lift the affected code out of C and into Python (for example, the import
> system rewrite).

There's a significant amount of wishful thinking here.  When CPython
code gets written in C, it's often because it's presumed to be
performance-critical.  The import system was indeed rewritten in
Python... but some parts were then written back in C (see
Python/import.c) to fix performance regressions.

Aye, I know, but the overall organisation of the import system still changed to "Python with C acceleration of key parts" rather than the previous "almost entirely C" approach.

Other implementations are now free to reuse the Python parts, and only have to implement the accelerators separately.

> In a lot of ways, CPython's C API can be viewed as one of the many
> competing approaches to enabling "C-with-objects" programming, just like
> C++.

It's a clumsy API to use *from C*.  Witness the inconsistent behaviour
of those APIs wrt. borrowed references, for example, or the fact that
forgetting an INCREF or DECREF can lead to occasional leaks or crashes,
or the delicate task of teaching the cyclic GC about a particular type,
or all the special cases where those APIs deviate slightly in semantics
from their pure Python equivalents, or the fact that APIs are added in
adhoc manner, leading to an inconsistent set of primitives...

Right, but that clumsiness isn't any easier to manage from C++ than it is from C. Things like RAII rely on consistent behaviour from the resources being managed, and the current C API doesn't offer that.

For example, putting a borrowed or stolen reference into an RAII based reference manager will result in an early free or a resource leak, just as happens when you decref a borrowed reference or incref a stolen one in C.

Hence the need for wrapper APIs and tools that make the C API more C++ friendly rather than it being straightforward to use directly.

CPython's C API is probably fine for consumption by intermediate layers
such as Cython. It's a maze to navigate for direct use.

That I also agree with, and hence I'd definitely be a fan if we ever figured out a way to bootstrap Cython into the CPython build process so that accelerated standard library modules could be written in Cython.

Unlike C++, Cython uses the same data structures and object model as Python does, and it already smooths over the irregularities in the C API to allow for fully automated resource management. The Python based syntax can even help lower the barriers to entry for folks that don't already know C (although you do still need to learn the underlying C memory model).