Better compatibility of the Python scientific/data stack with fast Python interpreters
Dear Numpy community members and Numpy developers, This email is to get the points of view of the Numpy community members and developers about a subject that I find very important. I'm going to introduce it just in few lines so I write few statements without backing them with proper arguments and without giving links and word definitions. I assume that most people registered to the numpy-discussion list are at least familiar to the subject. I think getting proper compatibility of the Python scientific/data stack with fast Python interpreters is very important for the long-term future of Python for science and data. Nowadays, fast Python implementations like PyPy and GraalPy are - complicated to use (need local builds of wheels) - slow as soon as the code involves Python extensions (because extensions work through an emulation layer, like cpyext in PyPy) Therefore, these fast Python implementations are not as popular as they should be. However, they are really great tools which should be used in particular for scientific/data applications. Unfortunately, if we just follow the pace of the CPython C API evolution, we will basically never have an ecosystem natively compatible with fast Python implementations. The more I think and learn about this subject, the more I think that Numpy has to stop using directly the CPython C API and to be ported to HPy (though there are other alternatives - in particular Nanobind and Cython - that could be discussed). Numpy 1 has been ported to HPy but unfortunately, there have been deep changes with Numpy 2 (and Meson) so it seems that one should restart the porting. Moreover, unfortunately, HPy does not currently receive as much care as it should. It seems to me that the project of fixing the roots of the Python ecosystem has to be relaunched. I think that the dynamics has to come from the Python scientific/data community and in particular Numpy. It is unfortunately outside of the C API working group's scope (see https://discuss.python.org/t/c-api-working-group-and-plan-to-get-a-python-c-...). It seems to me that it is necessary (and possible) to get some founding for such an impactful project so that we can get people working on it. I'd like to write a long and serious text on this subject, and I first try to get the points of view of the different people and projects involved. I guess I should write explicit questions: - What do you think about the project of fixing the Python scientific/data stack so that it becomes natively compatible (hence fast and convenient, with easy installations) with alternative and fast Python interpreters? - Do you have points of view on how this should be done, technically (HPy?, something else?) and on other aspects (community, NEP?, founding, ...). - Anything else interesting on this subject? Best regards, Pierre -- Pierre Augier - CR CNRS http://www.legi.grenoble-inp.fr LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels BP53, 38041 Grenoble Cedex, France tel:+33.4.56.52.86.16
On Tue, Apr 29, 2025 at 11:24 AM PIERRE AUGIER < pierre.augier@univ-grenoble-alpes.fr> wrote:
Dear Numpy community members and Numpy developers,
This email is to get the points of view of the Numpy community members and developers about a subject that I find very important. I'm going to introduce it just in few lines so I write few statements without backing them with proper arguments and without giving links and word definitions. I assume that most people registered to the numpy-discussion list are at least familiar to the subject.
Thanks for thinking about this Pierre.
I think getting proper compatibility of the Python scientific/data stack with fast Python interpreters is very important for the long-term future of Python for science and data.
I'm not sure it is, it wouldn't rank high on my wish list. PyPy is nearing its end-of-life, and GraalPy is really only for Java devs it seems - I've never seen it in the real world. With CPython performance being worked hard on, the performance gap is shrinking every year. More importantly, none of these efforts (including the "faster CPython" project), seem critical to numerical/scientific users. We're still talking about pure Python code that gets maybe up to 5x faster, while the gains from doing things in compiled languages are a lot higher. So the benefits are more important for small packages if it moves the threshold at which it becomes necessary for them to write zero extension modules. For core libraries like NumPy, pure Python performance isn't super critical. When thinking about overall performance improvements, I'd say that the two most promising large developments are: (1) making it easier to use accelerator libraries (PyTorch, CuPy et al.), and (2) free-threaded CPython for enabling Python-level threading.
Nowadays, fast Python implementations like PyPy and GraalPy are
- complicated to use (need local builds of wheels) - slow as soon as the code involves Python extensions (because extensions work through an emulation layer, like cpyext in PyPy)
Therefore, these fast Python implementations are not as popular as they should be. However, they are really great tools which should be used in particular for scientific/data applications.
Unfortunately, if we just follow the pace of the CPython C API evolution, we will basically never have an ecosystem natively compatible with fast Python implementations.
The more I think and learn about this subject, the more I think that Numpy has to stop using directly the CPython C API and to be ported to HPy (though there are other alternatives - in particular Nanobind and Cython - that could be discussed). Numpy 1 has been ported to HPy but unfortunately, there have been deep changes with Numpy 2 (and Meson) so it seems that one should restart the porting.
Moreover, unfortunately, HPy does not currently receive as much care as it should.
It seems to me that the project of fixing the roots of the Python ecosystem has to be relaunched. I think that the dynamics has to come from the Python scientific/data community and in particular Numpy. It is unfortunately outside of the C API working group's scope (see https://discuss.python.org/t/c-api-working-group-and-plan-to-get-a-python-c-... ).
It seems to me that it is necessary (and possible) to get some founding for such an impactful project so that we can get people working on it.
I'd like to write a long and serious text on this subject, and I first try to get the points of view of the different people and projects involved.
I guess I should write explicit questions:
- What do you think about the project of fixing the Python scientific/data stack so that it becomes natively compatible (hence fast and convenient, with easy installations) with alternative and fast Python interpreters? - Do you have points of view on how this should be done, technically (HPy?, something else?) and on other aspects (community, NEP?, founding, ...). - Anything else interesting on this subject?
Having HPy or something like it will be very nice and lead ot long-term benefits. The problem with HPy seems to be more a social one at this point: if CPython core devs don't want to adopt it but do their own "make the C API more opaque" strategy, then more effort on HPy isn't going to help. If you're going to dig into this more, I suggest trying to get a very good sense of what the CPython core dev team, and in particular its C API Workgroup, is thinking/planning. That will inform whether the right strategy is to help their efforts along, or work on HPy. Cheers, Ralf
Best regards, Pierre
-- Pierre Augier - CR CNRS http://www.legi.grenoble-inp.fr LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels BP53, 38041 Grenoble Cedex, France tel:+33.4.56.52.86.16 _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@googlemail.com
----- Mail original -----
De: "Ralf Gommers via NumPy-Discussion" <numpy-discussion@python.org> À: "Discussion of Numerical Python" <numpy-discussion@python.org> Cc: "hpy-dev" <hpy-dev@python.org>, "pypy-dev" <pypy-dev@python.org>, "ralf gommers" <ralf.gommers@gmail.com> Envoyé: Mercredi 30 Avril 2025 07:32:44 Objet: [Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
On Tue, Apr 29, 2025 at 11:24 AM PIERRE AUGIER < [ mailto:pierre.augier@univ-grenoble-alpes.fr | pierre.augier@univ-grenoble-alpes.fr ] > wrote:
Dear Numpy community members and Numpy developers,
This email is to get the points of view of the Numpy community members and developers about a subject that I find very important. I'm going to introduce it just in few lines so I write few statements without backing them with proper arguments and without giving links and word definitions. I assume that most people registered to the numpy-discussion list are at least familiar to the subject.
Thanks for thinking about this Pierre.
Thanks Ralf for your answer. Even if it is not very positive, it it VERY interesting!
I think getting proper compatibility of the Python scientific/data stack with fast Python interpreters is very important for the long-term future of Python for science and data.
I'm not sure it is, it wouldn't rank high on my wish list. PyPy is nearing its end-of-life, and GraalPy is really only for Java devs it seems - I've never seen it in the real world.
- PyPy and GraalPy are not used a lot because the Python ecosystem is highly incompatible with them since it is based on the CPython C API, which assumes in particular reference counting. This is really a chicken and egg problem. With a better ecosystem, we can anticipate a strong increase of PyPy and GraalPy usage. - I'm not aware that PyPy is nearing its end-of-life. It would really be a shame for Python to lose this project. - GraalPy is not at all only for Java devs. It can be used by every Python users. It is just a young project and its main issue for usability is the lack of specific wheels. Numpy implemented with HPy would fix that. However, Java devs can also be Python and Numpy users and getting Numpy fully compatible with GraalPy would be very good for applications using Java and Python.
With CPython performance being worked hard on, the performance gap is shrinking every year.
I don't think the word "shrinking" is very adapted :-) ! Even on pyperformance (which is a set of benchmarks designed for CPython development), PyPy and GraalPy are still typically 4 to 5 times faster than CPython. For pure Python CPU bounded cases, it is more of the order of 20 times faster, which is typically comparable with what one gets with NodeJS. Unfortunately, the Faster CPython project is getting slowly and has a very hard time to get significant performance improvements. One can have a look at https://github.com/faster-cpython/benchmarking-public. It would be a great achievement if CPython 3.14 with its JIT can be 1.5 faster than CPython 3.10 (measured with pyperformance, again compared to typically 4 to 5 times faster with PyPy and GraalPy). The constrains on CPython (in particular related to the CPython C API) are so strong that for most cases the Faster CPython project won't be able to reach what can be obtained with fast Python implementations using advanced methods and algorithms. As long as the CPython C API assumes reference counting (and other implementation details bad for performance), CPython performance cannot reached what can be obtained with PyPy, GraalPy or NodeJS. If CPython changes too much its internals, it would need an emulation layer like PyPy's cpyext, and then extensions would become slower, which is of course not acceptable.
More importantly, none of these efforts (including the "faster CPython" project), seem critical to numerical/scientific users. We're still talking about pure Python code that gets maybe up to 5x faster, while the gains from doing things in compiled languages are a lot higher.
For numerical kernels, specialized static languages are clearly needed, but for global organisation and orchestration, PyPy and GraalPy are really very efficient (I guess in most cases efficient enough). For pure Python code and compared to CPython, the factor is not 5x but more 10x or 20x.
So the benefits are more important for small packages if it moves the threshold at which it becomes necessary for them to write zero extension modules. For core libraries like NumPy, pure Python performance isn't super critical.
For libraries like NumPy, it is not. But for users of these libraries, it is. Or at least getting the possibility to use fast Python (typically as fast as Javascript) would change what we can do with Python and what can be coded in Python. For example, currently, it is not doable to use a lot of small Python objects in performance critical code so nice OOP Python cannot be used. With alternative fast Python implementations more usable with an ecosystem using HPy and universal wheels, it would become possible. I remember during a Python training a person (it was a good C++ developer) trying to write something doing ray tracing with OOP and Numpy. It was super slow with CPython so I told him to try in pure Python with PyPy (it was just a game to learn so why not). He soon got a prototype and was quite impressed with the speed. Then there was something else in the code that was convenient to be done with Numpy and then, pff, all the good performance disappeared because the Numpy calls where super slow (going through cpyext) and the rest of the code was then using some Numpy numbers. This was very frustrating. We also see projects using a lot Python (with lot of small objects) that are so slow compared to comparable alternatives (for example Pelican compared to Hugo in Go; myst-parser compared to mystmd in JavaScript; I also think about Sphinx and its so long builds, ...). It would change the life of Python users if it would be easy to use fast implementations with performance similar to NodeJS. Moreover, if boxing/unboxing can be avoided between Python JITs and native code (which should be doable with HPy), Numpy calls would be significantly faster. With Graal VM it could even be more spectacular because it could be possible that the same JIT compiles together Python and native code without barriers across the language boundaries.
When thinking about overall performance improvements, I'd say that the two most promising large developments are: (1) making it easier to use accelerator libraries (PyTorch, CuPy et al.), and (2) free-threaded CPython for enabling Python-level threading.
Good and usable sub-interpreters would also be great and I think this is highly coupled with Numpy-HPy.
[...]
Moreover, unfortunately, HPy does not currently receive as much care as it should.
It seems to me that the project of fixing the roots of the Python ecosystem has to be relaunched. I think that the dynamics has to come from the Python scientific/data community and in particular Numpy. It is unfortunately outside of the C API working group's scope (see [ https://discuss.python.org/t/c-api-working-group-and-plan-to-get-a-python-c-... | https://discuss.python.org/t/c-api-working-group-and-plan-to-get-a-python-c-... ] ).
It seems to me that it is necessary (and possible) to get some founding for such an impactful project so that we can get people working on it.
I'd like to write a long and serious text on this subject, and I first try to get the points of view of the different people and projects involved.
I guess I should write explicit questions:
- What do you think about the project of fixing the Python scientific/data stack so that it becomes natively compatible (hence fast and convenient, with easy installations) with alternative and fast Python interpreters? - Do you have points of view on how this should be done, technically (HPy?, something else?) and on other aspects (community, NEP?, founding, ...). - Anything else interesting on this subject?
Having HPy or something like it will be very nice and lead to long-term benefits. The problem with HPy seems to be more a social one at this point: if CPython core devs don't want to adopt it but do their own "make the C API more opaque" strategy, then more effort on HPy isn't going to help. If you're going to dig into this more, I suggest trying to get a very good sense of what the CPython core dev team, and in particular its C API Workgroup, is thinking/planning. That will inform whether the right strategy is to help their efforts along, or work on HPy.
This is very interesting that you write that! This is exactly what I tried to understand with my questions in https://discuss.python.org/t/c-api-working-group-and-plan-to-get-a-python-c-... I think that the discussion was very informative. It is in particular interesting to see that the C API working group is only focused (i) on the "historical" CPython C API (HPy is considered as an external project) and (ii) on slow and gradual evolution of this C API (to go at the pace of the slowest projects using the CPython C API). Some CPython core devs work in the direction of "making the C API more opaque", but important "revolutionary" changes (like providing an API not assuming reference counting) have just been discussed (https://github.com/capi-workgroup/api-revolution/issues) and it seems that there are not yet practical plans for them (for example https://github.com/markshannon/New-C-API-for-Python/ is stopped). Moreover, there are no time schedule and there cannot be any time schedule since the pace depends on all CPython C API users. In practice, to get in a reasonable time (not 12 years) the scientific/data Python stack natively compatible with advanced methods used in dynamic language interpreters, one needs (i) to invest on HPy and (ii) to port Numpy to something else than the CPython C API (it can be HPy but it could also be Cython or maybe other things). Also, it seems that the C API working group does not care about the principle of the HPy universal wheels (compatible across Python versions and implementations). To have something like that, we need HPy. It seems to me that a strategy based on HPy would give practical benefices for users in a much shorter time (typically few years) than just waiting for CPython C API evolution. Let's recall that Numpy 1 has already been ported in HPy (like Matplotlib by the way). I thus agree with
The problem with HPy seems to be more a social one at this point
Cheers, Pierre
Hello, I'd like to just add a few data points from my Gentoo experience. On Wed, 2025-04-30 at 07:32 +0200, Ralf Gommers via pypy-dev wrote:
On Tue, Apr 29, 2025 at 11:24 AM PIERRE AUGIER < pierre.augier@univ-grenoble-alpes.fr> wrote:
I think getting proper compatibility of the Python scientific/data stack with fast Python interpreters is very important for the long-term future of Python for science and data.
I'm not sure it is, it wouldn't rank high on my wish list. PyPy is nearing its end-of-life, [...]
I'm sorry but I don't understand what you're referring to. Sure, PyPy is not moving fast, but it definitely isn't dead. Sure, it sucks that we're still stuck in Python 3.11 era, but that doesn't make PyPy EOL.
More importantly, none of these efforts (including the "faster CPython" project), seem critical to numerical/scientific users. We're still talking about pure Python code that gets maybe up to 5x faster, while the gains from doing things in compiled languages are a lot higher. So the benefits are more important for small packages if it moves the threshold at which it becomes necessary for them to write zero extension modules. For core libraries like NumPy, pure Python performance isn't super critical.
A user has made an interesting argument once -- while there is no gain from using, say, NumPy on PyPy compared to CPython, there are big projects that happen to 1) have major performance gains in their pure Python code, and 2) use NumPy as a dependency. I'm not saying NumPy needs to necessarily be optimized for that use case, just pointing out that it happens to be used in the wild, and for good reasons. -- Best regards, Michał Górny
On Wed, Apr 30, 2025 at 3:30 PM Michał Górny <mgorny@gentoo.org> wrote:
Hello,
I'd like to just add a few data points from my Gentoo experience. ... I'm sorry but I don't understand what you're referring to. Sure, PyPy is not moving fast, but it definitely isn't dead. Sure, it sucks that we're still stuck in Python 3.11 era, but that doesn't make PyPy EOL.
-- Best regards, Michał Górny
It does not appear there will be a PyPy 3.12. When NumPy drops support for Python3.11 (Jan 2026), future versions based on the CPython C-API will defacto no longer be relevant for PyPy3.11. However, if HPy succeeds, there is a chance that a NumPy "universal" HPy wheel will still work on PyPy 3.11, as the API will be abstracted away from a direct connection with the CPython C-API. This is one of the attractions of HPy, that puts it in a different category than Cython or nanobind: universal HPy binaries can run on any python interpreter or version that supports the abstracted API. With the current level of active contributions to PyPy, I am not convinced we can help making HPy a reality, even though it is a very well thought out solution to once and for all detaching third party libraries from a tight integration with the CPython C-API. I agree HPy would be advantageous to both the CPython core team and package maintainers. The yearly churn of packages needing to update for new CPython C-API would be localized in the HPy layer for the new CPython (similar to Cython, nanobind, and PyO3), and the CPython maintainers would be free to change more aspects of the CPython internals that unintentionally impact the external C-API (which differentiates HPy from Cython, nanobind, and PyO3). But not all good ideas get to win out and become the popular, goto solution. PyPy itself is an example that, unfortunately. Matti
Thanks Matti for this interesting and sad piece of news. I also reply on numpy-discussion@python.org since your message was not post on this list and take the opportunity to signal another important post about his subject on discuss.python.org by Stepan Sindelar: https://discuss.python.org/t/c-api-working-group-and-plan-to-get-a-python-c-... IMHO, anyone interested in the long term future of Python should be aware of these issues. Pierre ----- Mail original -----
De: "matti picus via hpy-dev" <hpy-dev@python.org> À: "PyPy Developer Mailing List" <pypy-dev@python.org>, "Ralf Gommers" <rgommers@quansight.com>, "hpy-dev" <hpy-dev@python.org> Envoyé: Vendredi 2 Mai 2025 10:05:51 Objet: [hpy-dev] Re: [pypy-dev] Re: [Numpy-discussion] Better compatibility of the Python scientific/data stack with fast Python interpreters
On Wed, Apr 30, 2025 at 3:30 PM Michał Górny <mgorny@gentoo.org> wrote:
Hello,
I'd like to just add a few data points from my Gentoo experience. ... I'm sorry but I don't understand what you're referring to. Sure, PyPy is not moving fast, but it definitely isn't dead. Sure, it sucks that we're still stuck in Python 3.11 era, but that doesn't make PyPy EOL.
-- Best regards, Michał Górny
It does not appear there will be a PyPy 3.12. When NumPy drops support for Python3.11 (Jan 2026), future versions based on the CPython C-API will defacto no longer be relevant for PyPy3.11. However, if HPy succeeds, there is a chance that a NumPy "universal" HPy wheel will still work on PyPy 3.11, as the API will be abstracted away from a direct connection with the CPython C-API. This is one of the attractions of HPy, that puts it in a different category than Cython or nanobind: universal HPy binaries can run on any python interpreter or version that supports the abstracted API.
With the current level of active contributions to PyPy, I am not convinced we can help making HPy a reality, even though it is a very well thought out solution to once and for all detaching third party libraries from a tight integration with the CPython C-API. I agree HPy would be advantageous to both the CPython core team and package maintainers. The yearly churn of packages needing to update for new CPython C-API would be localized in the HPy layer for the new CPython (similar to Cython, nanobind, and PyO3), and the CPython maintainers would be free to change more aspects of the CPython internals that unintentionally impact the external C-API (which differentiates HPy from Cython, nanobind, and PyO3). But not all good ideas get to win out and become the popular, goto solution. PyPy itself is an example that, unfortunately. Matti _______________________________________________ hpy-dev mailing list -- hpy-dev@python.org To unsubscribe send an email to hpy-dev-leave@python.org https://mail.python.org/mailman3/lists/hpy-dev.python.org/ Member address: pierre.augier@univ-grenoble-alpes.fr
On 2025-05-02 10:05, matti picus via pypy-dev wrote:
It does not appear there will be a PyPy 3.12.
I'll expand a bit/add my POV/my rephrasing: Matti and I don't have the capacity to keep up with CPython's development on our own. If PyPy 3.12., 3.13 and 3.14 are supposed to happen, we'll need other contributors to become interested in working on these features. So far, we are not aware of any such person, but I'm certainly happy to support anyone who wants to go in that direction. If that's you, please contact us! That doesn't mean that PyPy is unsupported or dead. I'm using it actively to do research (e.g. for CPU simulation: https://arxiv.org/abs/2503.04389 ) and we have a number of features in development (improving JIT warmup, some Python optimizations, a sampling memory profiler). But CPython is unfortunately very active, and it's not really possible to stay up-to-date with new versions with just two active devs. Cheers, CF
Thanks Matti for this interesting and sad piece of news. I also reply on numpy-discussion@python.org since your message was not post on this list and take the opportunity to signal another important post about his subject on discuss.python.org by Stepan Sindelar: https://network4you.com
On 06/05/2025 14.23, CF Bolz-Tereick via pypy-dev wrote:
If PyPy 3.12., 3.13 and 3.14 are supposed to happen, we'll need other contributors to become interested in working on these features. So far, we are not aware of any such person, but I'm certainly happy to support anyone who wants to go in that direction. If that's you, please contact us!
Do you think it might help if there were a blog article sketching out the tasks involved? What is the process of updating to a new Python version? How did you do this in the past? How could things be split into more and less involved parts? E.g. you guys focusing on core functionality, while others work primarily on lifting the code base to newer Python versions and standard library. Is this similar to rebasing? Something like that. Optionally, follow up with nudging LWN or any other popular and relevant site.
----- Mail original -----
De: "matti picus via hpy-dev" <hpy-dev@python.org> À: "PyPy Developer Mailing List" <pypy-dev@python.org>, "Ralf Gommers" <rgommers@quansight.com>, "hpy-dev" <hpy-dev@python.org> Envoyé: Vendredi 2 Mai 2025 10:05:51 Objet: [hpy-dev] Re: [pypy-dev] Re: [Numpy-discussion] Better compatibility of the Python scientific/data stack with fast Python interpreters
On Wed, Apr 30, 2025 at 3:30 PM Michał Górny <mgorny@gentoo.org> wrote:
Hello,
I'd like to just add a few data points from my Gentoo experience. ... I'm sorry but I don't understand what you're referring to. Sure, PyPy is not moving fast, but it definitely isn't dead. Sure, it sucks that we're still stuck in Python 3.11 era, but that doesn't make PyPy EOL.
-- Best regards, Michał Górny
It does not appear there will be a PyPy 3.12. When NumPy drops support for Python3.11 (Jan 2026), future versions based on the CPython C-API will defacto no longer be relevant for PyPy3.11. However, if HPy succeeds, there is a chance that a NumPy "universal" HPy wheel will still work on PyPy 3.11, as the API will be abstracted away from a direct connection with the CPython C-API. This is one of the attractions of HPy, that puts it in a different category than Cython or nanobind: universal HPy binaries can run on any python interpreter or version that supports the abstracted API.
With the current level of active contributions to PyPy, I am not convinced we can help making HPy a reality, even though it is a very well thought out solution to once and for all detaching third party libraries from a tight integration with the CPython C-API. I agree HPy would be advantageous to both the CPython core team and package maintainers. The yearly churn of packages needing to update for new CPython C-API would be localized in the HPy layer for the new CPython (similar to Cython, nanobind, and PyO3), and the CPython maintainers would be free to change more aspects of the CPython internals that unintentionally impact the external C-API (which differentiates HPy from Cython, nanobind, and PyO3). But not all good ideas get to win out and become the popular, goto solution. PyPy itself is an example that, unfortunately. Matti
Hello, I'd like to try to work on this performance issue https://github.com/pypy/pypy/issues/5277. It is about the fact that allocations of types defined with HPy are very slow compared to the CPython counterpart (typically 7 times slower). I think it is a key problem when considering a potential revival of the HPy project with new funding (which would help PyPy). Some data in old messages seem to indicate that it is at least partly a regression: it seems that PyPy was in Sept. 2022 only something like 2 times slower. I guess fixing issue #5277 is very technical and I realize that I might not be able to be efficient on this subject. However, I'm motivated to learn and I can spend a bit of time and energy on this subject. So I'm asking for some kind of mentoring. Matti and Carl Friedrich, I would need your point of view on this project. Is it even possible to try to do something about #5277 for someone who has only a superficial knowledge of PyPy internals? Then, where and how to start? I guess the first thing to do would be to try to understand why PyPy is so slow by profiling (?), studying traces (?) and/or studying PyPy code, in particular the HPy implementation? I'd like to be clear that it is just a proposition and that I realize that I'm asking for your time. If you think that it does not make sense to try to work on this direction (#5277, HPy in general), please tell me! Pierre
Hi Pierre, so one thing you can definitely do without deep knowledge is bisect the performance degradation. We have nightly builds here: https://buildbot.pypy.org/nightly/ It's a little bit tedious work, but if it's such a clear slowdown, finding out what broke this is useful. For more involved dev work, maybe it makes sense for us to set up a video call and brainstorm a bit what you would be interested in and would make strategic sense. I'm a bit busy with teaching still, but we can probably find a slot. Send me a private mail if you want to do that? Cheers, CF On 2025-06-04 01:44, PIERRE AUGIER wrote:
Hello,
I'd like to try to work on this performance issue https://github.com/ pypy/pypy/issues/5277. It is about the fact that allocations of types defined with HPy are very slow compared to the CPython counterpart (typically 7 times slower).
I think it is a key problem when considering a potential revival of the HPy project with new funding (which would help PyPy).
Some data in old messages seem to indicate that it is at least partly a regression: it seems that PyPy was in Sept. 2022 only something like 2 times slower.
I guess fixing issue #5277 is very technical and I realize that I might not be able to be efficient on this subject. However, I'm motivated to learn and I can spend a bit of time and energy on this subject.
So I'm asking for some kind of mentoring. Matti and Carl Friedrich, I would need your point of view on this project. Is it even possible to try to do something about #5277 for someone who has only a superficial knowledge of PyPy internals?
Then, where and how to start? I guess the first thing to do would be to try to understand why PyPy is so slow by profiling (?), studying traces (?) and/or studying PyPy code, in particular the HPy implementation?
I'd like to be clear that it is just a proposition and that I realize that I'm asking for your time. If you think that it does not make sense to try to work on this direction (#5277, HPy in general), please tell me!
Pierre
----- Mail original -----
De: "Carl Friedrich Bolz-Tereick" <cfbolz@gmx.de> À: "PIERRE AUGIER" <pierre.augier@univ-grenoble-alpes.fr>, "PyPy Developer Mailing List" <pypy-dev@python.org> Envoyé: Jeudi 5 Juin 2025 09:53:42 Objet: Re: [hpy-dev] Re: [pypy-dev] Re: [Numpy-discussion] Better compatibility of the Python scientific/data stack with fast Python interpreters
Hi Pierre,
so one thing you can definitely do without deep knowledge is bisect the performance degradation. We have nightly builds here: https://buildbot.pypy.org/nightly/
It's a little bit tedious work, but if it's such a clear slowdown, finding out what broke this is useful.
Thanks for the advice. Interestingly, I can confirm that there was a clear slowdown in the hpy-0.0.4 branch: fbfb93b991eb is indeed 3 times slower than 5a77b3ed145a (which is as slow as current PyPy). $ hg log -r fbfb93b991eb changeset: 106205:fbfb93b991eb branch: hpy-0.0.4 parent: 106198:bda034f1cbe8 user: Matti Picus <matti.picus@gmail.com> date: Tue Oct 04 16:34:43 2022 +0300 summary: remove debug cruft $ hg log -r 5a77b3ed145a changeset: 106090:5a77b3ed145a branch: hpy-0.0.4 user: Matti Picus <matti.picus@gmail.com> date: Thu Sep 08 23:42:01 2022 +0300 summary: refactor to make ztranslation work, skip debug tests that use closed handles Unfortunately, I can't bisect more from https://buildbot.pypy.org/nightly/hpy-0.0.4/ since there are no intermediate nightly builds. There are 114 commits between 5a77b3ed145a and fbfb93b991eb and the changes are quite large so I guess it would be nice to bisect a bit more (?). However, it's going to be long if I need to build PyPy for each try. Would it be possible to ask the build bot to build few commits between 5a77b3ed145a and fbfb93b991eb? For the record, few commands to get the results: ~/tmp/pypy-hpy0.0.4-c-jit-106209-fbfb93b991eb-linux64/bin/pypy -m venv .venv_pypy-hpy0.0.4-fbfb93b991eb . .venv_pypy-hpy0.0.4-fbfb93b991eb/bin/activate pip install pytest rm -rf build/ *.so python setup.py --hpy-abi=universal build_ext --inplace pytest -v test_microbench.py::TestType::test_allocate_obj and for HPy microbenchmarks, I use commit 1789d59dd1fb with these changes diff --git a/microbench/setup.py b/microbench/setup.py --- a/microbench/setup.py +++ b/microbench/setup.py @@ -2,7 +2,7 @@ from setuptools import setup, Extension setup( name="hpy.microbench", - setup_requires=['hpy.devel', 'cffi'], + # setup_requires=['hpy.devel', 'cffi'], ext_modules = [ Extension('cpy_simple', ['src/cpy_simple.c'], diff --git a/microbench/src/hpy_simple.c b/microbench/src/hpy_simple.c --- a/microbench/src/hpy_simple.c +++ b/microbench/src/hpy_simple.c @@ -64,9 +64,7 @@ static HPy allocate_tuple_impl(HPyContex /* Foo type */ -typedef struct { - PyObject_HEAD -} FooObject; +typedef struct {} FooObject;
For more involved dev work, maybe it makes sense for us to set up a video call and brainstorm a bit what you would be interested in and would make strategic sense. I'm a bit busy with teaching still, but we can probably find a slot. Send me a private mail if you want to do that?
Cheers,
CF
On 2025-06-04 01:44, PIERRE AUGIER wrote:
Hello,
I'd like to try to work on this performance issue https://github.com/ pypy/pypy/issues/5277. It is about the fact that allocations of types defined with HPy are very slow compared to the CPython counterpart (typically 7 times slower).
I think it is a key problem when considering a potential revival of the HPy project with new funding (which would help PyPy).
Some data in old messages seem to indicate that it is at least partly a regression: it seems that PyPy was in Sept. 2022 only something like 2 times slower.
I guess fixing issue #5277 is very technical and I realize that I might not be able to be efficient on this subject. However, I'm motivated to learn and I can spend a bit of time and energy on this subject.
So I'm asking for some kind of mentoring. Matti and Carl Friedrich, I would need your point of view on this project. Is it even possible to try to do something about #5277 for someone who has only a superficial knowledge of PyPy internals?
Then, where and how to start? I guess the first thing to do would be to try to understand why PyPy is so slow by profiling (?), studying traces (?) and/or studying PyPy code, in particular the HPy implementation?
I'd like to be clear that it is just a proposition and that I realize that I'm asking for your time. If you think that it does not make sense to try to work on this direction (#5277, HPy in general), please tell me!
Pierre
On Fri, Jun 6, 2025 at 12:41 PM PIERRE AUGIER via pypy-dev <pypy-dev@python.org> wrote:
Thanks for the advice. Interestingly, I can confirm that there was a clear slowdown in the hpy-0.0.4 branch: fbfb93b991eb is indeed 3 times slower than 5a77b3ed145a (which is as slow as current PyPy).
$ hg log -r fbfb93b991eb changeset: 106205:fbfb93b991eb branch: hpy-0.0.4 parent: 106198:bda034f1cbe8 user: Matti Picus <matti.picus@gmail.com> date: Tue Oct 04 16:34:43 2022 +0300 summary: remove debug cruft
$ hg log -r 5a77b3ed145a changeset: 106090:5a77b3ed145a branch: hpy-0.0.4 user: Matti Picus <matti.picus@gmail.com> date: Thu Sep 08 23:42:01 2022 +0300 summary: refactor to make ztranslation work, skip debug tests that use closed handles
Nice detective work. The relevant git commit mappings from that branch to py3.11 are fbfb93b991eb -> 3379a71102495cb and 5a77b3ed145a -> 50099512d3077f. If I look at the diff from git diff 50099512d3077 3379a71102495cb -- pypy/module/_hpy_universal/interp_type.py there were a bunch of changes that might impact performance. Perhaps they can be refactored. Matti
participants (7)
-
CF Bolz-Tereick -
Marcus Harnisch -
matti picus -
Michał Górny -
PIERRE AUGIER -
Ralf Gommers -
Techran CO