[Python-Dev] Let's change to C API!

[Antoine Pitrou]

On Thu, 23 Aug 2018 13:34:02 +1200
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Neil Schemenauer wrote:
> > Perhaps a "argument clinic on steroids" would be the proper
> > approach.  So, extensions would mostly be written in C.  However, we
> > would have a pre-processor that does some "magic" to make using the
> > Python API cleaner.  
> 
> You seem to have started on the train of thought that
> led me to create Pyrex (the precursor to Cython).
> 
> It went like this: I was getting frustrated with SWIG
> because it wasn't powerful enough to do what I wanted.
> I thought about something that would let me write functions
> with Python headers and C bodies.
> 
> Then I realised that to really make the C API bearable it
> would need to take care of refcounting and exception handling,
> so the body couldn't just be plain C, it would need some Python
> semantics as well. Given that, it seemed more sensible to base
> all of the syntax on Python.
> 
> Also, I wanted to avoid the problem you get with all
> preprocessors, that when something goes wrong you get
> error messages in terms of the expanded code rather than
> the original source. To fix that I would need to do a
> full parsing and type analysis job -- essentially it
> would need to be a full-blown compiler in its own right.
> 
> So, "argument clinic on steroids" is actually a pretty
> good description of Pyrex.

Agreed, at least abstractly, Cython is the best existing solutiong to
the problem.

The issues to solve for Cython to be usable for the CPython stdlib are
IMHO the following:

- the bootstrap problem (Cython self-compiles with CPython)
- the dependency / versioning problem (Cython is a large quick-evolving
  third-party package that we can't decently vendor)
- the maintenance problem (how do ensure we can change small things in
  the C API, especially semi-private ones, without having to submit PRs
  to Cython as well)
- the debugging problem (Cython's generated C code is unreadable,
  unlike Argument Clinic's, which can make debugging annoying)

Regards

Antoine.