Future of HPy? Faster CPython, and comparison between different Python interpreters
Hi HPy developers,
I send this email here since it can reach people interested in HPy, and PyPy/GraalPy developers.
CPython 3.13 has an experimental JIT. The results in term of global performance can be followed here https://github.com/faster-cpython/benchmarking-public.
It is still very modest (best results are something like 1.4 time faster than CPython 3.10). However, there are other plans to improve CPython new JIT (https://github.com/faster-cpython/ideas/blob/main/3.14/README.md, see also https://github.com/faster-cpython/ideas/issues/701#issuecomment-2405979384) so we can guess that CPython 3.14 and 3.15 will be (a bit?) faster.
In https://github.com/faster-cpython/ideas/blob/main/3.14/README.md, I read with interest that HPy is mentioned!
#### HPy-like C API
Tagged integers will require new internal APIs to pass opaque references, so we might as well add a new, consistent API modeled on HPy.
We will use this internally to start with, but we expect tools like Cython and
Pybind11 will want to use it to avoid the overhead of going through the PyObject * based API.
So I was wondering about the state of HPy. For example, is there a plan to port Numpy 2 in HPy? What about HPy support in Cython? Do you still think HPy could become more popular and used?
Another related question is about the performance comparison between different interpreters. It seems to me that the difference of performance between CPython and other alternative interpreters is a strong argument for HPy.
Seeing the performance comparisons with different versions and commits of CPython, I was wondering about similar comparisons with other interpreters (in practice PyPy and GraalPy). It would be very interesting to know from fair benchmarks about the performance of different interpreters.
In https://github.com/oracle/graalpython, it's written "GraalPy is ~4x faster than CPython on the official Python Performance Benchmark Suite".
In https://pypy.org/, there is written "PyPy is 4.4 times faster than CPython 3.7", but comparing to CPython 3.7 does not make sense nowadays. Is it also measured with https://pyperformance.readthedocs.io/ ?
Do you think it would be interesting and feasible to have a website showing fair comparisons between different Python interpreters ?
I guess a simple solution would be to use faster-cpython workflows by asking them to also consider stable releases of PyPy and GraalPy. I'm going to ask in https://github.com/faster-cpython/ideas.
Would there be a good alternative solution to get such data?
Pierre Augier
Is it (PyPy) also measured with https://pyperformance.readthedocs.io/ ?
We run a different version of those benchmarks at speed.pypy.org, where you can also see a comparison to CPython 3.11 and CPython 3.12. Our benchmarks still run on Python2.7 (well, PyPy 2.7) since PyPy is written in RPython which is a python2.7 variant.
Do you think it would be interesting and feasible to have a website showing fair comparisons between different Python interpreters ?
Much ink has been spilled over what is "fair". The different interpreters have different warmup characteristics. Speed is not always the only factor: some people care about memory usage or other parameters. And even getting consistent benchmarks over time is difficult. If the goal is to track performance over time in a sandboxed consistent environment, I think there is value in the proposal. But a one-off "hey X is N.NN faster than Y" is going to be problematic.
I will let others answer as for HPy. Matti
On Fri, Nov 15, 2024 at 5:32 PM PIERRE AUGIER <pierre.augier@univ-grenoble-alpes.fr> wrote:
Thank you Matti for your answer.
----- Mail original -----
IMHO, it's a bit unfortunate that this implementation detail has such consequence. It would be better if the different Python implementations can work together on the same benchmark tool (I guess https://pyperformance.readthedocs.io/, maybe adapted to be more friendly with alternative implementations).
Yes, one number is clearly insuffisant to summarize the performance of an implementation. I still think that we have an issue about the communication on the performance comparison between different Python implementations, which is bad for the global understanding of the community that users would benefit to have an ecosystem more open to alternative implementations. First, there are different websites for the benchmarks of the different implementations with (slightly) different benchmarks, methodologies and bases for comparison:
- https://pypy.org/ and https://speed.pypy.org/ for PyPy (based on PyPy benchmarks, compatible with Python 2.7)
- https://github.com/faster-cpython/benchmarking-public for CPython (based on https://github.com/python/pyperformance)
- A figure on https://github.com/oracle/graalpython for GraalPy (based on https://github.com/python/pyperformance)
- https://speed.python.org/ (not up-to-date, complicated and not very readable figures)
Second, it is very difficult with what we have to go beyond comparisons with 1 number "X is N.NN faster than Y", especially for people that do not know the details of pyperformance benchmarks. It would be much better to be able to present something much more quantitative with few numbers associated with groups of benchmarks for different types of applications (for example, Pure Python, numerics with C extensions, startup, web servers, ...). For some cases, it would also be useful to have simple comparisons with other technologies (based on Python or on another language).
Of course, I realize that I just point out a difficulty and do not bring any solution, but it might be a first step to improve the situation.
Is it (PyPy) also measured with https://pyperformance.readthedocs.io/ ?
We run a different version of those benchmarks at speed.pypy.org, where you can also see a comparison to CPython 3.11 and CPython 3.12. Our benchmarks still run on Python2.7 (well, PyPy 2.7) since PyPy is written in RPython which is a python2.7 variant.
Do you think it would be interesting and feasible to have a website showing fair comparisons between different Python interpreters ?
Much ink has been spilled over what is "fair". The different interpreters have different warmup characteristics. Speed is not always the only factor: some people care about memory usage or other parameters. And even getting consistent benchmarks over time is difficult. If the goal is to track performance over time in a sandboxed consistent environment, I think there is value in the proposal. But a one-off "hey X is N.NN faster than Y" is going to be problematic.
I will let others answer as for HPy. Matti
On Fri, Nov 15, 2024 at 5:32 PM PIERRE AUGIER <pierre.augier@univ-grenoble-alpes.fr> wrote:
Thank you Matti for your answer.
----- Mail original -----
IMHO, it's a bit unfortunate that this implementation detail has such consequence. It would be better if the different Python implementations can work together on the same benchmark tool (I guess https://pyperformance.readthedocs.io/, maybe adapted to be more friendly with alternative implementations).
Yes, one number is clearly insuffisant to summarize the performance of an implementation. I still think that we have an issue about the communication on the performance comparison between different Python implementations, which is bad for the global understanding of the community that users would benefit to have an ecosystem more open to alternative implementations. First, there are different websites for the benchmarks of the different implementations with (slightly) different benchmarks, methodologies and bases for comparison:
- https://pypy.org/ and https://speed.pypy.org/ for PyPy (based on PyPy benchmarks, compatible with Python 2.7)
- https://github.com/faster-cpython/benchmarking-public for CPython (based on https://github.com/python/pyperformance)
- A figure on https://github.com/oracle/graalpython for GraalPy (based on https://github.com/python/pyperformance)
- https://speed.python.org/ (not up-to-date, complicated and not very readable figures)
Second, it is very difficult with what we have to go beyond comparisons with 1 number "X is N.NN faster than Y", especially for people that do not know the details of pyperformance benchmarks. It would be much better to be able to present something much more quantitative with few numbers associated with groups of benchmarks for different types of applications (for example, Pure Python, numerics with C extensions, startup, web servers, ...). For some cases, it would also be useful to have simple comparisons with other technologies (based on Python or on another language).
Of course, I realize that I just point out a difficulty and do not bring any solution, but it might be a first step to improve the situation.
participants (2)
-
matti picus -
PIERRE AUGIER