Mailman 3 June 2016 - pypy-dev

About gurobipy
by Luis José Novoa 31 May '18

31 May '18

Hi All, My name is Luis Novoa. Im working on a large scale optimization problem using python/gurobipy for my implementation. I was wondering if any of you in this list has used gurobipy with pypy, or if it is even possible at this time. Thanks in advance, Luis.

9 9

Twisted's deferToThread is much more slower with pypy than on cpython
by Nagy, Attila 18 Jul '16

18 Jul '16

Hi, Consider this example program: https://gist.github.com/bra-fsn/1fd481b44590a939e849cb9073ba1a41 # pypy /tmp/deft.py 1 deferToThread: avg 1310.90 us, sync: avg 1.13 us, 1155.26x increase deferToThread: avg 248.42 us, sync: avg 0.19 us, 1288.78x increase deferToThread: avg 288.20 us, sync: avg 0.20 us, 1450.80x increase deferToThread: avg 366.34 us, sync: avg 0.20 us, 1860.96x increase deferToThread: avg 300.38 us, sync: avg 0.20 us, 1510.52x increase deferToThread: avg 341.94 us, sync: avg 0.20 us, 1751.63x increase deferToThread: avg 282.96 us, sync: avg 0.20 us, 1432.18x increase deferToThread: avg 232.87 us, sync: avg 0.20 us, 1185.64x increase deferToThread: avg 350.51 us, sync: avg 0.20 us, 1751.06x increase deferToThread: avg 379.09 us, sync: avg 0.20 us, 1903.13x increase While this runs, it uses only around 15% CPU. Running this on more threads yields even worse results: # pypy /tmp/deft.py 8 deferToThread: avg 8236.71 us, sync: avg 0.28 us, 29359.01x increase deferToThread: avg 8286.90 us, sync: avg 0.32 us, 26029.29x increase deferToThread: avg 8345.67 us, sync: avg 0.67 us, 12436.32x increase deferToThread: avg 8685.53 us, sync: avg 0.33 us, 26617.55x increase deferToThread: avg 8689.54 us, sync: avg 0.48 us, 18093.90x increase deferToThread: avg 8743.51 us, sync: avg 0.31 us, 28124.92x increase deferToThread: avg 8760.89 us, sync: avg 0.32 us, 27270.95x increase deferToThread: avg 8757.82 us, sync: avg 0.32 us, 27791.52x increase deferToThread: avg 5884.83 us, sync: avg 0.22 us, 26213.80x increase deferToThread: avg 5927.90 us, sync: avg 0.23 us, 25773.31x increase deferToThread: avg 6410.40 us, sync: avg 0.23 us, 27943.53x increase deferToThread: avg 6133.28 us, sync: avg 0.23 us, 26417.04x increase deferToThread: avg 6182.15 us, sync: avg 0.23 us, 26529.44x increase deferToThread: avg 6679.73 us, sync: avg 0.22 us, 30022.54x increase deferToThread: avg 6390.30 us, sync: avg 0.22 us, 28647.95x increase deferToThread: avg 6509.35 us, sync: avg 0.23 us, 28201.91x increase deferToThread: avg 6640.61 us, sync: avg 0.23 us, 28499.73x increase deferToThread: avg 6508.86 us, sync: avg 0.23 us, 28293.32x increase deferToThread: avg 6394.18 us, sync: avg 0.23 us, 27980.45x increase deferToThread: avg 6584.74 us, sync: avg 0.23 us, 28983.66x increase deferToThread: avg 6806.19 us, sync: avg 0.24 us, 28535.81x increase The process now only uses around 2% CPU. Running on even more threads: # pypy /tmp/deft.py 16 deferToThread: avg 27107.54 us, sync: avg 0.29 us, 94991.56x increase deferToThread: avg 28288.76 us, sync: avg 0.29 us, 96298.54x increase deferToThread: avg 28365.01 us, sync: avg 3.86 us, 7346.72x increase deferToThread: avg 28927.80 us, sync: avg 0.33 us, 86955.17x increase deferToThread: avg 28958.93 us, sync: avg 0.29 us, 101428.44x increase deferToThread: avg 29037.54 us, sync: avg 0.30 us, 98424.06x increase deferToThread: avg 29223.17 us, sync: avg 0.29 us, 99730.33x increase deferToThread: avg 29270.21 us, sync: avg 0.37 us, 78388.17x increase deferToThread: avg 29312.24 us, sync: avg 0.32 us, 90785.14x increase deferToThread: avg 29462.58 us, sync: avg 0.29 us, 101798.22x increase deferToThread: avg 29475.06 us, sync: avg 0.32 us, 91309.64x increase deferToThread: avg 29618.17 us, sync: avg 0.28 us, 105142.35x increase deferToThread: avg 29781.02 us, sync: avg 0.32 us, 93284.17x increase deferToThread: avg 29986.18 us, sync: avg 3.21 us, 9327.75x increase deferToThread: avg 30258.60 us, sync: avg 0.42 us, 71364.92x increase deferToThread: avg 30525.20 us, sync: avg 0.28 us, 107570.08x increase This eats between 0.2-1% CPU. And now on cypthon: # python /tmp/deft.py 1 deferToThread: avg 316.17 us, sync: avg 1.38 us, 228.71x increase deferToThread: avg 312.92 us, sync: avg 1.38 us, 226.96x increase deferToThread: avg 320.22 us, sync: avg 1.39 us, 230.37x increase deferToThread: avg 317.33 us, sync: avg 1.35 us, 235.24x increase # python /tmp/deft.py 8 deferToThread: avg 2542.90 us, sync: avg 1.37 us, 1854.14x increase deferToThread: avg 2544.50 us, sync: avg 1.35 us, 1878.13x increase deferToThread: avg 2544.47 us, sync: avg 1.36 us, 1864.52x increase deferToThread: avg 2544.52 us, sync: avg 1.38 us, 1839.01x increase deferToThread: avg 2544.92 us, sync: avg 1.36 us, 1871.81x increase deferToThread: avg 2546.71 us, sync: avg 1.39 us, 1830.35x increase deferToThread: avg 2552.38 us, sync: avg 1.35 us, 1893.17x increase deferToThread: avg 2552.40 us, sync: avg 1.36 us, 1870.20x increase # python /tmp/deft.py 16 deferToThread: avg 4745.76 us, sync: avg 1.26 us, 3770.11x increase deferToThread: avg 4748.67 us, sync: avg 1.24 us, 3817.03x increase deferToThread: avg 4749.81 us, sync: avg 1.26 us, 3756.39x increase deferToThread: avg 4749.72 us, sync: avg 1.24 us, 3839.88x increase deferToThread: avg 4749.87 us, sync: avg 1.28 us, 3709.99x increase deferToThread: avg 4752.63 us, sync: avg 1.24 us, 3842.90x increase deferToThread: avg 4752.53 us, sync: avg 1.23 us, 3866.08x increase deferToThread: avg 4752.55 us, sync: avg 1.23 us, 3855.40x increase deferToThread: avg 4754.03 us, sync: avg 1.29 us, 3678.09x increase deferToThread: avg 4754.97 us, sync: avg 1.25 us, 3817.19x increase deferToThread: avg 4755.45 us, sync: avg 1.32 us, 3593.28x increase deferToThread: avg 4756.35 us, sync: avg 1.25 us, 3804.18x increase deferToThread: avg 4756.19 us, sync: avg 1.29 us, 3687.73x increase deferToThread: avg 4757.19 us, sync: avg 1.23 us, 3860.74x increase deferToThread: avg 4758.02 us, sync: avg 1.24 us, 3824.33x increase deferToThread: avg 4759.63 us, sync: avg 1.24 us, 3830.40x increase The interpreter uses around 100% CPU. Python 2.7.11 PyPy 5.1.1 FreeBSD 10/amd64 Why deferToThread is so slow is another question, but here I'm interested in pypy's abysmal performance.

2 3

pickle numpy array from pypy to cpython?
by Eli Stevens (Gmail) 16 Jul '16

16 Jul '16

I'm trying to construct some data that includes numpy arrays in pypy, pickle it, then unpickle it in cpython (to use some non-pypy-compatible libs). However, the actual class of the pickled array is _numpypy.multiarray, which cpython doesn't have. Any suggestions? Thanks, Eli

6 23

Update to the win32 buildbot
by Matti Picus 10 Jul '16

10 Jul '16

Hi! I am cc-ing the pypy-dev list so there is some permanent record of this, the mail is for the win32 buildbot maintainer. Could you add the lzma development libraries to the build environment of the win32 buildbot? They are needed for python 3, as you can see from any of the failing build runs like this one http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/2398/steps/s… Python 3 uses the xz-5.0.5 libraries, as can bee seen in the cpython/PCbuild/readme.txt, following their links leads to http://tukaani.org/xz/ and to a download http://tukaani.org/xz/xz-5.0.5-windows.zip (library) http://tukaani.org/xz/xz-5.0.5-windows.zip.sig (signature) I will try to update the local.zip available on our downloads https://bitbucket.org/pypy/pypy/downloads/local_2.4.zip and the documentation http://doc.pypy.org/en/latest/windows.html but in the meantime it would be great if you could get this going Thanks! Matti

2 1

Is it possible to make PyPy smaller?
by Paul Baker 30 Jun '16

30 Jun '16

I have a small command-line tool, written in Python, which I would like to give to a few people to use on their own machines. For performance reasons, I would like to distribute the tool with PyPy rather than CPython. For this particular tool, PyPy runs 60 times faster than CPython (and only 3 times slower than a less-capable C++ implementation). My concern with distributing PyPy to end-users is size: Python27.dll is 2.3MB while libpypy-c.dll is 41.9MB. Why is PyPy so much bigger than CPython? Is it possible to build a version of PyPy which is closer to CPython in size?

3 3

Passing an array to C in RFFI
by John Zhang 29 Jun '16

29 Jun '16

Hi all, Thanks for the help previously. I have another question regarding passing an array value to C function using RFFI. Suppose I have a function in C that’s like this: void fn(void** array_of_voips, int length) How do I in RPython call the function using RFFI passing an array value for it? it seems that I can’t just do fn([a, b, c], 3) Help appreciated. John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang(a)anu.edu.au<mailto:john.zhang@anu.edu.au>

2 2

Recursive struct definition in rffi?
by John Zhang 26 Jun '16

26 Jun '16

Hi guys, I’m using RFFI to reflect a C API interface that my colleague has developed. During this process I encountered the problem of not knowing how to define a recursive struct definition. e.g. How do I define a struct like the following: typedef struct A A; struct A { … void (*fn) (A*); } Help appreciated. Regards, John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang(a)anu.edu.au<mailto:john.zhang@anu.edu.au>

2 1

Any core devs want to come to the python compilers workshop next month?
by Nathaniel Smith 26 Jun '16

26 Jun '16

Hi all, This is super short notice but... you may be aware that I'm organizing a workshop next month (July 11-12) in Austin, for some of the different Python compiler projects to get together and compare notes on how best to achieve world domination... or at least make the Python ecosystem more friendly to non-CPython implementations: https://python-compilers-workshop.github.io/ Originally Maciej and Matti were going to represent PyPy, but unfortunately Maciej had to cancel. Matti will still be there, so that's awesome, but I find myself with a chunk of unused sponsorship $$, and given how PyPy is currently by far the most successful of these projects, I feel like it'd be even more awesome if we had even more representation from the PyPy core devs :-). So if any of you are interested in a free last minute trip to Austin + a chance to maybe help influence future plans and funding in a more PyPy-friendly direction, please get in touch ASAP... -n -- Nathaniel J. Smith -- https://vorpus.org

2 2

Micro NumPy & VecOpt 2.0
by Richard Plangger 25 Jun '16

25 Jun '16

Hi, in case you have not heard: I'm currently working on the PPC and S390X port for micro numpy. Thanks to IBM for funding this work. I'm ~50% through the ppc operations to implement. The goal is to turn this optimization on (by default) in the micro numpy module. I recently had the idea to enhance the jit driver by giving it more information about parallel execution. I'm *not* talking about the main interp. loop. Having a vectorized loop that executes parallel in threads would certainly push micronumpy performance. Has somebody already tried something similar? I think it is a challenge, but it should be possible (with a reasonable amount of work) to get a simple thread fork/join model such as OpenMP provides. Cheers, Richard

1 0

Implementation of some Python 3.5 features for more PyPy 3.5 support
by Raffael Tfirst 24 Jun '16

24 Jun '16

Hi, I am Raffael Tfirst, currently working on PyPy 3.5 for the Google Summer of Code. Since it might be of interest to some, I want to give you a short summary of what I do. The Python 3.5 features I implement in PyPy are the @ operator for matrix multiplications (PEP 465), additional unpacking generalizations (PEP 448) and coroutines with async and await syntax (PEP 492). A short description of each feature can be found here (at New Syntax Features): https://docs.python.org/3/whatsnew/3.5.html It would be cool to have those features implemented in PyPy because they will most likely be used a lot in future Python 3.5 programs, that is why I want to close this gap and push PyPy a bit closer to Python 3.5 functionality. I started working on this about a month ago. As things stand now, the @ operator already works. The additional unpacking feature will be ready soon, too, since a great part is already implemented. There's just some nasty errors I need to fix before it can be tested. In about two weeks progress will speed up a lot, because then I'm going to have way more time for this project. For those who are interested, I work on the py3.5 branch. You can also check out my progress on my blog: http://pypy35syntax.blogspot.co.at/

4 3