On 11/20/2018 10:33 PM, Nathaniel Smith wrote:
On Tue, Nov 20, 2018 at 6:05 PM Glenn Linderman <v+python@g.nevcal.com> wrote:
On 11/20/2018 2:17 PM, Victor Stinner wrote:
IMHO performance and hiding implementation details are exclusive. You
should either use the C API with impl. details for best performances,
or use a "limited" C API for best compatibility.
The "limited" C API concept would seem to be quite sufficient for extensions that want to extend Python functionality to include new system calls, etc. (pywin32, pyMIDI, pySide, etc.) whereas the numpy and decimal might want best performance.
To make things more complicated: numpy and decimal are in a category
of modules where if you want them to perform well on JIT-based VMs,
then there's no possible C API that can achieve that. To get the
benefits of a JIT on code using numpy or decimal, the JIT has to be
able to see into their internals to do inlining etc., which means they
can't be written in C at all [1], at which point the C API becomes
irrelevant.

It's not clear to me how this affects any of the discussion in
CPython, since supporting JITs might not be part of the goal of a new
C API, and I'm not sure how many packages fall between the
numpy/decimal side and the pure-ffi side.

-n

[1] Well, there's also the option of teaching your Python JIT to
handle LLVM bitcode as a source language, which is the approach that
Graal is experimenting with. It seems completely wacky to me to hope
you could write a C API emulation layer like PyPy's cpyext, and
compile that + C extension code to LLVM bitcode, translate the LLVM
bitcode to JVM bytecode, inline the whole mess into your Python JIT,
and then fold everything away to produce something reasonable. But I
could be wrong, and Oracle is throwing a lot of money at Graal so I
guess we'll find out.

Interesting, thanks for the introduction to wacky. I was quite content with the idea that numpy, and other modules that would choose to use the unlimited API, would be sacrificing portability to non-CPython implementations... except by providing a Python equivalent (decimal, and some others do that, IIRC).

Regarding JIT in general, though, it would seem that "precompiled" extensions like numpy would not need to be re-compiled by the JIT. 

But if it does, then the JIT better understand/support C syntax, but JVM JITs probably don't! so that leads to the scenario you describe.