Mailman 3 July 2016 - pypy-dev

About gurobipy
by Luis José Novoa 31 May '18

31 May '18

Hi All, My name is Luis Novoa. Im working on a large scale optimization problem using python/gurobipy for my implementation. I was wondering if any of you in this list has used gurobipy with pypy, or if it is even possible at this time. Thanks in advance, Luis.

9 9

NumPyPy vs NumPy
by Papa, Florin 05 Aug '16

05 Aug '16

Hi, This is Florin Papa from the Dynamic Scripting Languages Optimizations Team at Intel Corporation. I have been working with NumPyPy to evaluate its performance and it seems significantly slower compared to CPython NumPy or even PyPy NumPy (installed with pip). The results were gathered after running microbenchmarks inspired from here: http://www.labri.fr/perso/nrougier/teaching/numpy.100/ These benchmarks perform basic tasks, such as matrix multiplication, generate Cauchy matrix, generate Gaussian array, find min and max in matrix, convert float array to integer array in place, sum all elements in array. Please see below the results containing run time, normalized to the CPython NumPy results (baseline). Benchmark CPython NumPy PyPy NumPy PyPy NumPyPy cauchy 1 5.838852812 4.866947551 pointbypoint 1 4.922654347 0.981008211 numrand 1 2.478997019 1.082185897 rowmean 1 2.512893263 1.062233015 dsums 1 33.58240465 1.013388981 vectsum 1 1.738446611 0.771660704 cauchy 1 2.168377906 0.887388291 polarcoords 1 1.030962402 0.500905427 vectsort 1 2.214586698 0.973727924 arange 1 2.045342386 0.69941044 vectoradd 1 5.447667037 1.513217941 extractint 1 1.655717606 2.671712185 float2int 1 3.1688 0.905406988 insertzeros 1 2.375043445 1.037504453 Is there an official benchmark suite for NumPy or a more relevant workload to compare against CPython? What is NumPyPy's maturity / adoption rate from your knowledge? The benchmarks used to collect the results are attached in this mail. Regards, Florin

6 13

how to extend VTUNE support from pypy for application hot spot analysis
by Wang, Peter Xihong 04 Aug '16

04 Aug '16

Hi, Armin created a minimum working version incorporating VTUNE to a branch at: https://bitbucket.org/pypy/pypy/branch/vtune This works in a sense such that statistically meaningful micro-architecture data could be collected. However, for hot spot analysis, we'd like to trace back to the Python application source code. We know the VTUNE hook point, but don't know how or where to start from the rpython source code. I ran gdb against pypy binary and found it really hard to get to the entry point/or knowledge we wanted. Could anyone give us some hints? Thanks, Peter

3 11

PyPy3: Translation issue: "TyperError: arithmetic not supported on <INT>, its size is too small"
by David Naylor 24 Jul '16

24 Jul '16

Hi While translating the latest version of pypy3 on FreeBSD (and after applying the fix for https://bitbucket.org/pypy/pypy/issues/2348) I get the following error: File "/usr/local/home/dbn/ports/ports/lang/pypy3/work/pypy3.3-v5.2.0-alpha1- src/rpython/rtyper/rint.py", line 25, in opprefix self.lowleveltype) [translation:ERROR] TyperError: arithmetic not supported on <INT>, its size is too small .. (pypy.module.time.interp_time:988)clock .. block@3 with 2 exits(v1573) .. v1575 = eq(value_25, v1574) The error occurs when translating pypy/module/time/interp_time, line 996: if value == rffi.cast(clock_t, -1): Although this can be easily fixed by casting value to r_int instead, it then causes another issue on line 969: cpu_time = float(tms.c_tms_utime + tms.c_tms_stime) Again, same issue of TyperError with <INT> (this time, instead of with clock_t [1][2] but with TMS.tms_utime [3]). I have been unable to figure out how a rpython.rtyper.lltypesystem.lltype.Number (which a believe an INT is) gets converted into a rpython.rtyper.rint.IntegerRepr, or why it gets converted without an opprefix set. Any suggestions how to fix this issue, or what the root cause is, will be greatly appreciated. Regards David [1] clock_t is defined, in platcheck_73 as: -+- clock_t size: 4 unsigned: 0 --- [2] On FreeBSD clock_t is typedef as unsigned long. [3] TMS is defined in platcheck_7 as: -+- TMS align: 4 size: 16 fldofs tms_utime: 0 fldsize tms_utime: 4 fldunsigned tms_utime: 0 fldofs tms_stime: 4 fldsize tms_stime: 4 fldunsigned tms_stime: 0 fldofs tms_cutime: 8 fldsize tms_cutime: 4 fldunsigned tms_cutime: 0 fldofs tms_cstime: 12 fldsize tms_cstime: 4 fldunsigned tms_cstime: 0

2 1

Re: [pypy-dev] Microsoft is starting its own JIT for Python.
by Nathaniel Smith 21 Jul '16

21 Jul '16

It's not "Microsoft", it's an experimental side project of two people who happen to work at Microsoft. As I understand it, the core motivation is to explore how far they can get in dragging upstream CPython in a JITwards direction -- so it's written as a extension module for CPython, and it requires a very recent version of CPython (no support for python 2!). The advantage is that it's *extremely* compatible with CPython -- unlike pypy, it handles the latest version of the language, it has the same garbage collection rules, and not only do they get essentially perfect extension module compatibility, they get it with zero speed penalty. They can also potentially explore different compilation strategies (e.g. function at a time) that aren't as easy a fit to pypy's meta-tracing architecture, and can use it as a lever to push CPython to start adding features like dict versioning. The disadvantage versus PyPy obviously is that it's nowhere near as mature, there are reasons to doubt that their optimization strategy could ever be competitive with PyPy's given the need for CPython compatibility, and PyPy could conceivably catch up on the relevant axes (better faster cpyext, python 3 support, etc.) -- though this isn't a given either. Right now I think it makes more sense to think of it as an interesting experiment to watch than as a serious product that one hopes to use. In any case, one can't really get mad at some individuals for investing their spare time in the wrong thing. This is definitely not some strategic initiative from Microsoft-the-company. -n On Jul 21, 2016 3:16 PM, "Phyo Arkar" <phyo.arkarlwin(a)gmail.com> wrote: Hello PyPy Team; I guess you guys already known. Microsft is starting its own pypy : https://github.com/Microsoft/Pyjion The main reason they mention there is due to weak CPyext support but It is just duplicating work, as you guys are already working on it . Shouldn't microsoft contribute into CPyext work in pypy instead? Regards, Phyo. _______________________________________________ pypy-dev mailing list pypy-dev(a)python.org https://mail.python.org/mailman/listinfo/pypy-dev

1 0

Microsoft is starting its own JIT for Python.
by Phyo Arkar 21 Jul '16

21 Jul '16

Hello PyPy Team; I guess you guys already known. Microsft is starting its own pypy : https://github.com/Microsoft/Pyjion The main reason they mention there is due to weak CPyext support but It is just duplicating work, as you guys are already working on it . Shouldn't microsoft contribute into CPyext work in pypy instead? Regards, Phyo.

2 1

Twisted's deferToThread is much more slower with pypy than on cpython
by Nagy, Attila 18 Jul '16

18 Jul '16

Hi, Consider this example program: https://gist.github.com/bra-fsn/1fd481b44590a939e849cb9073ba1a41 # pypy /tmp/deft.py 1 deferToThread: avg 1310.90 us, sync: avg 1.13 us, 1155.26x increase deferToThread: avg 248.42 us, sync: avg 0.19 us, 1288.78x increase deferToThread: avg 288.20 us, sync: avg 0.20 us, 1450.80x increase deferToThread: avg 366.34 us, sync: avg 0.20 us, 1860.96x increase deferToThread: avg 300.38 us, sync: avg 0.20 us, 1510.52x increase deferToThread: avg 341.94 us, sync: avg 0.20 us, 1751.63x increase deferToThread: avg 282.96 us, sync: avg 0.20 us, 1432.18x increase deferToThread: avg 232.87 us, sync: avg 0.20 us, 1185.64x increase deferToThread: avg 350.51 us, sync: avg 0.20 us, 1751.06x increase deferToThread: avg 379.09 us, sync: avg 0.20 us, 1903.13x increase While this runs, it uses only around 15% CPU. Running this on more threads yields even worse results: # pypy /tmp/deft.py 8 deferToThread: avg 8236.71 us, sync: avg 0.28 us, 29359.01x increase deferToThread: avg 8286.90 us, sync: avg 0.32 us, 26029.29x increase deferToThread: avg 8345.67 us, sync: avg 0.67 us, 12436.32x increase deferToThread: avg 8685.53 us, sync: avg 0.33 us, 26617.55x increase deferToThread: avg 8689.54 us, sync: avg 0.48 us, 18093.90x increase deferToThread: avg 8743.51 us, sync: avg 0.31 us, 28124.92x increase deferToThread: avg 8760.89 us, sync: avg 0.32 us, 27270.95x increase deferToThread: avg 8757.82 us, sync: avg 0.32 us, 27791.52x increase deferToThread: avg 5884.83 us, sync: avg 0.22 us, 26213.80x increase deferToThread: avg 5927.90 us, sync: avg 0.23 us, 25773.31x increase deferToThread: avg 6410.40 us, sync: avg 0.23 us, 27943.53x increase deferToThread: avg 6133.28 us, sync: avg 0.23 us, 26417.04x increase deferToThread: avg 6182.15 us, sync: avg 0.23 us, 26529.44x increase deferToThread: avg 6679.73 us, sync: avg 0.22 us, 30022.54x increase deferToThread: avg 6390.30 us, sync: avg 0.22 us, 28647.95x increase deferToThread: avg 6509.35 us, sync: avg 0.23 us, 28201.91x increase deferToThread: avg 6640.61 us, sync: avg 0.23 us, 28499.73x increase deferToThread: avg 6508.86 us, sync: avg 0.23 us, 28293.32x increase deferToThread: avg 6394.18 us, sync: avg 0.23 us, 27980.45x increase deferToThread: avg 6584.74 us, sync: avg 0.23 us, 28983.66x increase deferToThread: avg 6806.19 us, sync: avg 0.24 us, 28535.81x increase The process now only uses around 2% CPU. Running on even more threads: # pypy /tmp/deft.py 16 deferToThread: avg 27107.54 us, sync: avg 0.29 us, 94991.56x increase deferToThread: avg 28288.76 us, sync: avg 0.29 us, 96298.54x increase deferToThread: avg 28365.01 us, sync: avg 3.86 us, 7346.72x increase deferToThread: avg 28927.80 us, sync: avg 0.33 us, 86955.17x increase deferToThread: avg 28958.93 us, sync: avg 0.29 us, 101428.44x increase deferToThread: avg 29037.54 us, sync: avg 0.30 us, 98424.06x increase deferToThread: avg 29223.17 us, sync: avg 0.29 us, 99730.33x increase deferToThread: avg 29270.21 us, sync: avg 0.37 us, 78388.17x increase deferToThread: avg 29312.24 us, sync: avg 0.32 us, 90785.14x increase deferToThread: avg 29462.58 us, sync: avg 0.29 us, 101798.22x increase deferToThread: avg 29475.06 us, sync: avg 0.32 us, 91309.64x increase deferToThread: avg 29618.17 us, sync: avg 0.28 us, 105142.35x increase deferToThread: avg 29781.02 us, sync: avg 0.32 us, 93284.17x increase deferToThread: avg 29986.18 us, sync: avg 3.21 us, 9327.75x increase deferToThread: avg 30258.60 us, sync: avg 0.42 us, 71364.92x increase deferToThread: avg 30525.20 us, sync: avg 0.28 us, 107570.08x increase This eats between 0.2-1% CPU. And now on cypthon: # python /tmp/deft.py 1 deferToThread: avg 316.17 us, sync: avg 1.38 us, 228.71x increase deferToThread: avg 312.92 us, sync: avg 1.38 us, 226.96x increase deferToThread: avg 320.22 us, sync: avg 1.39 us, 230.37x increase deferToThread: avg 317.33 us, sync: avg 1.35 us, 235.24x increase # python /tmp/deft.py 8 deferToThread: avg 2542.90 us, sync: avg 1.37 us, 1854.14x increase deferToThread: avg 2544.50 us, sync: avg 1.35 us, 1878.13x increase deferToThread: avg 2544.47 us, sync: avg 1.36 us, 1864.52x increase deferToThread: avg 2544.52 us, sync: avg 1.38 us, 1839.01x increase deferToThread: avg 2544.92 us, sync: avg 1.36 us, 1871.81x increase deferToThread: avg 2546.71 us, sync: avg 1.39 us, 1830.35x increase deferToThread: avg 2552.38 us, sync: avg 1.35 us, 1893.17x increase deferToThread: avg 2552.40 us, sync: avg 1.36 us, 1870.20x increase # python /tmp/deft.py 16 deferToThread: avg 4745.76 us, sync: avg 1.26 us, 3770.11x increase deferToThread: avg 4748.67 us, sync: avg 1.24 us, 3817.03x increase deferToThread: avg 4749.81 us, sync: avg 1.26 us, 3756.39x increase deferToThread: avg 4749.72 us, sync: avg 1.24 us, 3839.88x increase deferToThread: avg 4749.87 us, sync: avg 1.28 us, 3709.99x increase deferToThread: avg 4752.63 us, sync: avg 1.24 us, 3842.90x increase deferToThread: avg 4752.53 us, sync: avg 1.23 us, 3866.08x increase deferToThread: avg 4752.55 us, sync: avg 1.23 us, 3855.40x increase deferToThread: avg 4754.03 us, sync: avg 1.29 us, 3678.09x increase deferToThread: avg 4754.97 us, sync: avg 1.25 us, 3817.19x increase deferToThread: avg 4755.45 us, sync: avg 1.32 us, 3593.28x increase deferToThread: avg 4756.35 us, sync: avg 1.25 us, 3804.18x increase deferToThread: avg 4756.19 us, sync: avg 1.29 us, 3687.73x increase deferToThread: avg 4757.19 us, sync: avg 1.23 us, 3860.74x increase deferToThread: avg 4758.02 us, sync: avg 1.24 us, 3824.33x increase deferToThread: avg 4759.63 us, sync: avg 1.24 us, 3830.40x increase The interpreter uses around 100% CPU. Python 2.7.11 PyPy 5.1.1 FreeBSD 10/amd64 Why deferToThread is so slow is another question, but here I'm interested in pypy's abysmal performance.

2 3

pickle numpy array from pypy to cpython?
by Eli Stevens (Gmail) 16 Jul '16

16 Jul '16

I'm trying to construct some data that includes numpy arrays in pypy, pickle it, then unpickle it in cpython (to use some non-pypy-compatible libs). However, the actual class of the pickled array is _numpypy.multiarray, which cpython doesn't have. Any suggestions? Thanks, Eli

6 23

integration with java
by Andrey Rubik 16 Jul '16

16 Jul '16

Hi guys! I'am new at pypy development and need help. I want do integration between pypy and jemeter (this project on java: http://jmeter.apache.org/). As i learned on jemeter dev list, i need some pypy-java bindings. It may be some .jar file for example. Where i can get some pypy-java binding (.jar file)? Thanks in advance! -- Best Regards, Andrey Rubik

5 8

Update to the win32 buildbot
by Matti Picus 10 Jul '16

10 Jul '16

Hi! I am cc-ing the pypy-dev list so there is some permanent record of this, the mail is for the win32 buildbot maintainer. Could you add the lzma development libraries to the build environment of the win32 buildbot? They are needed for python 3, as you can see from any of the failing build runs like this one http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/2398/steps/s… Python 3 uses the xz-5.0.5 libraries, as can bee seen in the cpython/PCbuild/readme.txt, following their links leads to http://tukaani.org/xz/ and to a download http://tukaani.org/xz/xz-5.0.5-windows.zip (library) http://tukaani.org/xz/xz-5.0.5-windows.zip.sig (signature) I will try to update the local.zip available on our downloads https://bitbucket.org/pypy/pypy/downloads/local_2.4.zip and the documentation http://doc.pypy.org/en/latest/windows.html but in the meantime it would be great if you could get this going Thanks! Matti

2 1