[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Glenn Linderman]

On 11/20/2018 10:33 PM, Nathaniel Smith wrote:
> On Tue, Nov 20, 2018 at 6:05 PM Glenn Linderman <v+python at g.nevcal.com> wrote:
>> On 11/20/2018 2:17 PM, Victor Stinner wrote:
>>> IMHO performance and hiding implementation details are exclusive. You
>>> should either use the C API with impl. details for best performances,
>>> or use a "limited" C API for best compatibility.
>> The "limited" C API concept would seem to be quite sufficient for extensions that want to extend Python functionality to include new system calls, etc. (pywin32, pyMIDI, pySide, etc.) whereas the numpy and decimal might want best performance.
> To make things more complicated: numpy and decimal are in a category
> of modules where if you want them to perform well on JIT-based VMs,
> then there's no possible C API that can achieve that. To get the
> benefits of a JIT on code using numpy or decimal, the JIT has to be
> able to see into their internals to do inlining etc., which means they
> can't be written in C at all [1], at which point the C API becomes
> irrelevant.
>
> It's not clear to me how this affects any of the discussion in
> CPython, since supporting JITs might not be part of the goal of a new
> C API, and I'm not sure how many packages fall between the
> numpy/decimal side and the pure-ffi side.
>
> -n
>
> [1] Well, there's also the option of teaching your Python JIT to
> handle LLVM bitcode as a source language, which is the approach that
> Graal is experimenting with. It seems completely wacky to me to hope
> you could write a C API emulation layer like PyPy's cpyext, and
> compile that + C extension code to LLVM bitcode, translate the LLVM
> bitcode to JVM bytecode, inline the whole mess into your Python JIT,
> and then fold everything away to produce something reasonable. But I
> could be wrong, and Oracle is throwing a lot of money at Graal so I
> guess we'll find out.
>
Interesting, thanks for the introduction to wacky. I was quite content 
with the idea that numpy, and other modules that would choose to use the 
unlimited API, would be sacrificing portability to non-CPython 
implementations... except by providing a Python equivalent (decimal, and 
some others do that, IIRC).

Regarding JIT in general, though, it would seem that "precompiled" 
extensions like numpy would not need to be re-compiled by the JIT.

But if it does, then the JIT better understand/support C syntax, but JVM 
JITs probably don't! so that leads to the scenario you describe.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181120/f154403e/attachment.html>