GSoC'21 Blog: Improve performance through use of Pythran
Hi everyone: I’m Xingyu Liu, this year’s SciPy GSoC student :) This summer, I’m very fortunate to work on improving SciPy algorithms’ performance via Pythran, under the supervision of my awesome mentors Ralf Gommers and Serge Guelton! I’m so happy to share the blog of our work: http://serge-sans-paille.github.io/pythran-stories/gsoc21-improve-performanc... It has been a great experience working on this project in GSoC'21, and much thanks to the SciPy community for this opportunity and help on my project! Special thanks to my mentors, Ralf and Serge, who provided immense support for me to get through the difficulties! I have learnt a lot this summer and I’ll continue to contribute to SciPy and Pythran in the future!!! Cheers, Xingyu
On Mon, Aug 30, 2021 at 7:19 PM Xingyu Liu <xingyuliu@g.harvard.edu> wrote:
Hi everyone:
I’m Xingyu Liu, this year’s SciPy GSoC student :) This summer, I’m very fortunate to work on improving SciPy algorithms’ performance via Pythran, under the supervision of my awesome mentors Ralf Gommers and Serge Guelton! I’m so happy to share the blog of our work: http://serge-sans-paille.github.io/pythran-stories/gsoc21-improve-performanc...
Thanks for sharing Xingyu! This has been a really fun and productive GSoC project from my perspective. It definitely helped mature Pythran and its integration into SciPy. There are a number of open PRs left that are close to completion and that we'll get merged over the coming weeks, however overall a lot was already merged into both SciPy and Pythran as Xingyu's blog post shows. One of the main take-aways we learned recently was that we should use Pythran to accelerate the private part of a function, but leave the public signature and input validation in Python. This is something that wasn't obvious at the start of Xingyu's project, and we'll need to make sure we do that also for already-merged PRs (for consistently, and possibly to fix some undetected corner case bugs in master) before branching off 1.8.x. Other than that I think we are still pretty happy with Pythran; it gives us fast performance that's easier to obtain than with Cython. It has been a great experience working on this project in GSoC'21, and much
thanks to the SciPy community for this opportunity and help on my project! Special thanks to my mentors, Ralf and Serge, who provided immense support for me to get through the difficulties! I have learnt a lot this summer and I’ll continue to contribute to SciPy and Pythran in the future!!!
It's been a pleasure working with you too, and I'm happy you're going to continue contributing Xingyu! Cheers, Ralf
Great job Xingyu! And happy to read that you are planning to contribute more.
On 30.08.2021, at 20:37, Ralf Gommers <ralf.gommers@gmail.com> wrote:
One of the main take-aways we learned recently was that we should use Pythran to accelerate the private part of a function, but leave the public signature and input validation in Python. This is something that wasn't obvious at the start of Xingyu's project, and we'll need to make sure we do that also for already-merged PRs (for consistently, and possibly to fix some undetected corner case bugs in master) before branching off 1.8.x. Other than that I think we are still pretty happy with Pythran; it gives us fast performance that's easier to obtain than with Cython.
Ok, then I understand from this that we should do Pythran over Cython for new code. Just a thought. If we are going this road, it looks like we could have a Python API and a Pythran/(Cython) API. A user could then decide to skip the validation like that. Having the same signature would be less of a work in terms of doc as we could just add a note in the Python function about the existence of a Pythran version. Or we allow differences and clearly state that this API is not as strict as the Python API. Cheers, Pamphile
On Tue, Aug 31, 2021 at 11:10 AM Pamphile Roy <roy.pamphile@gmail.com> wrote:
Great job Xingyu! And happy to read that you are planning to contribute more.
On 30.08.2021, at 20:37, Ralf Gommers <ralf.gommers@gmail.com> wrote:
One of the main take-aways we learned recently was that we should use Pythran to accelerate the private part of a function, but leave the public signature and input validation in Python. This is something that wasn't obvious at the start of Xingyu's project, and we'll need to make sure we do that also for already-merged PRs (for consistently, and possibly to fix some undetected corner case bugs in master) before branching off 1.8.x. Other than that I think we are still pretty happy with Pythran; it gives us fast performance that's easier to obtain than with Cython.
Ok, then I understand from this that we should do Pythran over Cython for new code.
It's not as simple as that, it depends on what you're trying to do. If Pythran fits, like for individual kernels, then probably yes. Cython can do more complex things though, like interact with the Python and NumPy C APIs, threading, etc. Serge and I gave a talk at SciPy'21 that addresses this: https://www.youtube.com/watch?v=6a9D9WL6ZjQ&t=2s.
Just a thought. If we are going this road, it looks like we could have a Python API and a Pythran/(Cython) API. A user could then decide to skip the validation like that. Having the same signature would be less of a work in terms of doc as we could just add a note in the Python function about the existence of a Pythran version. Or we allow differences and clearly state that this API is not as strict as the Python API.
There's no such thing as a Pythran API. Pythran generates a Python API, and there's no separate C-like API. So whether a Python function uses Pythran under the hood should always be an implementation detail. It is unrelated to potentially skipping validation. Cheers, Ralf
Thanks for the precisions Ralf! Cheers, Pamphile
On 31.08.2021, at 11:51, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Tue, Aug 31, 2021 at 11:10 AM Pamphile Roy <roy.pamphile@gmail.com <mailto:roy.pamphile@gmail.com>> wrote: Great job Xingyu! And happy to read that you are planning to contribute more.
On 30.08.2021, at 20:37, Ralf Gommers <ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>> wrote:
One of the main take-aways we learned recently was that we should use Pythran to accelerate the private part of a function, but leave the public signature and input validation in Python. This is something that wasn't obvious at the start of Xingyu's project, and we'll need to make sure we do that also for already-merged PRs (for consistently, and possibly to fix some undetected corner case bugs in master) before branching off 1.8.x. Other than that I think we are still pretty happy with Pythran; it gives us fast performance that's easier to obtain than with Cython.
Ok, then I understand from this that we should do Pythran over Cython for new code.
It's not as simple as that, it depends on what you're trying to do. If Pythran fits, like for individual kernels, then probably yes. Cython can do more complex things though, like interact with the Python and NumPy C APIs, threading, etc. Serge and I gave a talk at SciPy'21 that addresses this: https://www.youtube.com/watch?v=6a9D9WL6ZjQ&t=2s <https://www.youtube.com/watch?v=6a9D9WL6ZjQ&t=2s>.
Just a thought. If we are going this road, it looks like we could have a Python API and a Pythran/(Cython) API. A user could then decide to skip the validation like that. Having the same signature would be less of a work in terms of doc as we could just add a note in the Python function about the existence of a Pythran version. Or we allow differences and clearly state that this API is not as strict as the Python API.
There's no such thing as a Pythran API. Pythran generates a Python API, and there's no separate C-like API. So whether a Python function uses Pythran under the hood should always be an implementation detail. It is unrelated to potentially skipping validation.
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (3)
-
Pamphile Roy -
Ralf Gommers -
Xingyu Liu