[Python-Dev] Let's change to C API!
solipsis at pitrou.net
Tue Jul 31 07:55:45 EDT 2018
On Tue, 31 Jul 2018 12:51:23 +0200
Victor Stinner <vstinner at redhat.com> wrote:
> 2018-07-31 8:58 GMT+02:00 Antoine Pitrou <solipsis at pitrou.net>:
> > What exactly in the C API made it slow or non-promising?
> >> The C API requires that your implementations make almost all the same
> >> design choices that CPython made 25 years ago (C structures, memory
> >> allocators, reference couting, specific GC implementation, GIL, etc.).
> > Yes, but those choices are not necessarily bad.
> I understood that PyPy succeeded to become at least 2x faster than
> CPython by stopping to use reference counting internally.
"I understood that"... where did you get it from? :-)
> I also want to make the debug build usable.
So I think that we should ask what the ABI differences between debug
and non-debug builds are.
AFAIK, the two main ones are Py_TRACE_REFS and Py_REF_DEBUG. Are there
Honestly, I don't think Py_TRACE_REFS is useful. I don't remember
any bug being discovered thanks to it. Py_REF_DEBUG is much more
useful. The main ABI issue with Py_REF_DEBUG is not object structure
(it doesn't change object structure), it's when a non-debug extension
steals a reference (or calls a reference-stealing C API function),
because then increments and decrements are unbalanced.
> I also want to allow OS vendors to provide multiple Python versions
> per OS release: *reduce* the maintenance burden, obviously it will
> still mean more work. It's a tradeoff depending on the lifetime of
> your OS and the pressure of customers to get the newest Python :-) FYI
> Red Hat already provide recent development tools on top of RHEL (and
> Centos and Fedora) because customers are asking for that. We don't
> work for free :-)
OS vendors seem to be doing a fine job AFAICT. And if I want a recent
Python I just download Miniconda/Anaconda.
> I also want to see more alternatives implementations of Python! I
> would like to see RustPython succeed!
As long as RustPython gets 10 commits a year, it has no chance of being
a functional Python implementation, let alone a successful one. AFAICS
it's just a toy project.
> > and one where I think Stefan is right that we
> > should push people towards Cython and alternatives, rather than direct
> > use of the C API (which people often fail to use correctly, in my
> > experience).
> Don't get me wrong: my intent is not to replace Cython. Even if PyPy
> is pushing hard cffi, many C extensions still use the C API.
cffi is a ctypes replacement. It's nice when you want to bind with
foreign C code, not if you want tight interaction with CPython objects.
> Maybe if the C API becomes more annoying and require developers to
> adapt their old code base for the "new C API", some of them will
> reconsider to use Cython, cffi or something else :-D
I think you don't realize that the C API is *already* annoying. People
started with it mostly because there wasn't a better alternative at the
time. You don't need to make it more annoying than it already is ;-)
Replacing existing C extensions with something else is entirely a
developer time/effort problem, not an attractivity problem. And I'm
not sure that porting a C extension to a new C API is more reasonable
than porting to Cython entirely.
> Do you think that it's wrong to promise that a smaller C API without
> implementation details will allow to more easily *experiment*
I don't think it's wrong. Though as long as CPython itself uses the
internal C API, you'll still have a *lot* of code to change before you
can even launch a functional interpreter and standard library...
It's just that I disagree that removing the C API will make CPython 2x
Actually, important modern optimizations for dynamic languages (such as
inlining, type specialization, inline caches, object unboxing) don't
seem to depend on the C API at all.
> >> I have to confess that helping Larry is part of my overall plan.
> > Which is why I'd like to see Larry chime in here.
> I already talked a little bit with Larry about my plan, but he wasn't
> sure that my plan is enough to be able to stop reference counting
> internally and move to a different garbage collector. I'm only sure
> that it's possible to keep using reference counting for the C API,
> since there are solutions for that (ex: maintain a hash table
> PyObject* => reference count).
Theoretically possible, but the cost of reference counting will go
through the roof if you start using a hash table.
> Honestly, right now, I'm only convinvced of two things:
> * Larry implementation is very complex and so I doubt that he is going
> to succeed. I'm talking about solutions to maintain optimize reference
> counting in multithreaded applications. Like his idea of "logs" of
> reference counters.
Well, you know, *any* solution is going to be very complex. Switching
to a full GC for a runtime (CPython) which can allocate hundreds of
thousands of objects per second will require a lot of optimization work
> * We have to change the C API: it causes troubles to *everybody*.
> Nobody spoke up because changing the C API is a giant project and it
> breaks the backward compatibility. But I'm not sure that all victims
> of the C API are aware that their issues are caused by the design of
> the current C API.
I fully agree that the C API is not very nice to play with. The
diversity of calling / error return conventions is one annoyance.
Borrowed references and reference stealing is another. Getting
reference counting right on all code paths is often delicate.
So I'm all for sanitizing the C API, and slowly deprecating old
patterns. And I think we should push people towards Cython for most
current uses of the C API.
More information about the Python-Dev