Mailman 3 January 2021 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer 21 Dec '22

21 Dec '22

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Some web benchmarks
by Oliver Margetts 26 Jan '21

26 Jan '21

Hello all, not sure where to post this, so I'll put it here. We recently wrote a blog post where we benchmarked some python web frameworks: https://suade.org/dev/12-requests-per-second-with-python/ One of the big conclusions was: if you want faster code, use pypy! I was pleasantly surprised as I was under the impression that web wasn't pypy's strongest area (I'm not sure where from). I wondered if you would consider adding something as simple as a "Hello, World" benchmark to speed.pypy.org? Aside from that, thanks for all of your efforts on making pypy fast!

3 3

Results NBabel benchmark CO2 production versus time: good new for PyPy map-improvements
by PIERRE AUGIER 26 Jan '21

26 Jan '21

Hi, I did some timing and energy consumption measurements with https://www.grid5000.fr/w/Energy_consumption_monitoring_tutorial I think the results tend to validate the approach used in the branch map-improvements (https://foss.heptapod.net/pypy/pypy/-/tree/branch/map-improvements). I attach one of the first figure including a run using an interpreter build with these changes (http://buildbot.pypy.org/nightly/map-improvements-3.7/pypy-c-jit-latest-lin…). To be compared with https://raw.githubusercontent.com/paugier/nbabel/master/py/fig/fig_ecolo_im… taken from Zwart (2020). The implementation run with PyPy map-improvements is faster than the implementations using Numba, Fortran and C++ (with flags like -Ofast and -march=native activated!) ! It's a great result! Congratulation! For people interested, the code for the benchmarks and the measurement is here https://github.com/paugier/nbabel Pierre

2 2

Hi. Help pls. Embedding PyPy in C++ but tkinter gives an error
by yaşar tecer 21 Jan '21

21 Jan '21

I Embedding PyPy in C++. Everything works fine. But when I import the tkinter module, I get an error. I am compiling the code in visual studio 2019. Sample code: #include <PyPy.h>#include <stdio.h> int main(){ int res; rpython_startup_code(); pypy_setup_home("C:\\pypy37\\",1); res = pypy_execute_source("import math ; print (math.pow(2,3)) "); pypy_init_threads(); return res; } PS C:\Users\.....\x64\Debug> ./pypyrun.exe8.0 The above code works without errors. But, the code below fails. The imported tkinter module is generating an error. #include <PyPy.h>#include <stdio.h> int main(){ int res; rpython_startup_code(); pypy_setup_home("C:\\pypy37\\",1); res = pypy_execute_source("from tkinter import *;window = Tk();window.mainloop() "); pypy_init_threads(); return res; } The error output is as follows; PS C:\Users\.....\x64\Debug> ./pypyrun.exe debug: OperationError: debug: operror-type: IndexError debug: operror-value: list index out of range

1 0

Freelist in PyPy? Reuse short lived objects?
by PIERRE AUGIER 21 Jan '21

21 Jan '21

Hello, I was still playing with the idea to speedup codes using small numerical objects. I wrote a Cython extension which defines a Point (3d) cdef class and a Points cdef class (a vector of points). Both classes contain a pointer towards a point_ C struct: ctypedef struct point_: float x, y, z Of course, any computation with Point objects with involved several very short lived objects and we really want to avoid all the associated malloc/free calls. In Cython, one can decorate a cdef class with `(a)cython.freelist(8)` to reused objects: https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#… I try to add a bit of logic to avoid freeing and allocating the memory for the struct (https://github.com/paugier/nbabel/blob/master/py/microbench/util_cython.pyx). If I understand correctly, doing such things is possible in CPython because the method __dealloc__ is called as soon as the objects is not accessible from Python. Or we can use the fact that it's very fast to get the reference count of an instance. But I think it is not the case for PyPy. Is there an alternative strategy efficient with PyPy? Pierre

3 5

Contributing Polyhedral Optimisations in PyPy
by muke101 19 Jan '21

19 Jan '21

I'm doing a computer science masters and am looking for an appropriate project to take on for a dissertation related to Polyhedral optimisations. Talking to my professor, we both think trying to implement the model and it's loop transformations in PyPy's JIT optimiser could be a good project to pursue, but before committing to anything I wanted to run this idea by the devs here who might be able to point out any hurdles I'd be likely to quickly come across that could prove difficult to solve at just a masters level, or whether or not these optimisations are actually already implemented in the first place (I have tried to google if this is the case and hadn't found anything, but can't be sure). I think this could have some good real world impact too as a lot of scientific code is written in Python and run on PyPy, and the Polyhedral model can offer substantial performance improvements in the form of auto-parallelization for these types of codes, which is why I'm interested in working on this for PyPy rather than CPython, although if anyone has good reason that I might want to look at CPython for this over PyPy please let me know. Appreciate any and all advice, thanks.

4 6

New Python/PyPy extension for object oriented numerically intensive codes ?
by PIERRE AUGIER 10 Jan '21

10 Jan '21

Hello, I wish you a Happy New Year. I would be very interested in using PyPy for numerically intensive scientific codes. I conclude from my experiments that PyPy could potentially be a great tool but that it is strongly limited in this area by the lack of some features in Python. For example: - Standard Python classes and standard Python instances are too dynamic for what we need for high performance. There should be a way to define in Python less dynamic classes and instances. - There is no specialized container to gather a fixed number of homogeneous objects and when possible to store them efficiently as native arrays of native variables. - There is no way to locally disable type and bound checks. I guess we don't have that in Python because it wouldn't be very useful for CPython and because Python has not been designed for performance. However, with efficient interpreters, I think many users would be happy to trade a bit of dynamism for better performance. I thought about what was missing and it seems to me that it could be provided by a Python/PyPy extension without addition to the Python language. However, Numpy API is IMHO not adapted. Now that Python is (and will be more and more) compared to Julia, I think it becomes necessary to have a good tool to write efficient numerical codes in pure Python style. I present here https://github.com/paugier/nbabel/blob/master/py/vector.md a possible new extension providing what I think would be needed to express in OOP Python things reasonable in terms of high performance computing. The detail of the proposed API is of course not very interesting at this point (it is just a dream). I am more interested about the point of view of PyPy developers and PyPy users about (i) the principle of this project (a Python extension to express in OOP Python things easier to be accelerated than standard Python) and (ii) about the technical feasibility of this project: Is it technically possible to extend Python and PyPy to develop such extension and make it very efficient? Which tools should be used? How should it be written? Best regards, Pierre

2 4

PyPy.org 2021 redesign suggestion
by Panos Laganakos 06 Jan '21

06 Jan '21

Hello, I was going over the PyPy website the other day, and something kept bugging me. While the project itself is in great condition really amazing at what it does, the website felt that it wasn't giving that information to the average visitor. So, I took a stab at it, with a basic mockup: https://www.dropbox.com/s/hnabjxy4aybfp1y/pypy.org-0.3.png?dl=0 And here is an annotated version: https://www.dropbox.com/s/r817xirkhcm9dks/pypy.org-0.3-annotations.png?dl=0 (hover over the image to see the annotations) And an announcement header one: https://www.dropbox.com/s/3fnvjcdm4aak7ys/pypy.org-0.3-announcement.png?dl=0 Let me know what you think. -- Pãnoș https://panoslaganakos.com

4 6

Funding for M1
by Christian Calderon 05 Jan '21

05 Jan '21

Hello PyPy team, If you haven't acquired funding for an M1 machine yet I'd be happy to make a donation to cover the cost. Just let me know the exact amount you need and what method you would like to receive the funding through. Thanks for all the good work! ~ Christian Calderon

1 0