Mailman 3 February 2015 - pypy-dev

use as benchmark pypy vs python if you please
by Dima Tisnek Sept. 13, 2015

Sept. 13, 2015

Attached is n-queens solver (pardon my naive algorithm), it runs: python 2.7.6: 17s pypy 2.4.0 alpha: 23s same nojit: 32s I've tried similar-looking algorithm for another problem before, and has similar results -- somehow pypy was slower. feel free to investigate / tweak or even use on speed.pypy.org

4 4

Poor performance for Krakatau
by Robert Grosse Aug. 10, 2015

Aug. 10, 2015

In some cases, Pypy performs worse than CPython when running Krakatau, and it has gotten a lot worse from 2.3 to 2.5 My benchmark is https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (for future reference, commit d889c7f44723e6d66a3630681f0385f173317dc9) You can run it with pypy Krakatau\benchmark.py -path <path to Java rt.jar> I get the following timings CPython: 17.43s Pypy 2.3: 38.48s (build pypy-c-jit-71056-c8e3b8cbc843-win32) Pypy 2.4: 39.92s (build pypy-c-jit-72200-375133966c12-win32) Pypy 2.5: 49.94s (build pypy-c-jit-74404-9fd586fe0fe5-win32) Not only was it slower than CPython to begin with, but it got much worse recently.

3 4

Windows: pypyw.exe?
by Yaacov Finkelman May 22, 2015

May 22, 2015

Hi, I was wondering what the plan is for fixing: https://bitbucket.org/pypy/pypy/issue/1651/ It is just a hassle with each upgrade of pypy. I'd be happy to fix, but I don't know where to start. Best, Jacob

4 8

fdopen(fd, 'w') whe fd is opened 'r'
by Matti Picus March 11, 2015

March 11, 2015

2 3

Vectorizing numpy traces
by Richard Plangger March 1, 2015

March 1, 2015

hi, i'm currently trying to find a way to generate simd code for python loops that operate on numpy arrays. The IRC is quite busy so I think I'd rather post it as an email... I have familiarized myself with pypy and read most of the docs and read the code in the metainterp to understand how the traces are built. I translated pypy with --withmod-micronumpy enabled (installed numpy pypy fork in an virtual env) and gave the jitviewer a try. Here is a sample program that does not really make sense, but I think it would contain opportunity to generate SIMD instructions. ``` import numpy def invoke(): a = numpy.zeros(1024) b = numpy.zeros(1024) c = numpy.zeros(1024) # (2) for i in range(0,1024): c[i] = a[i] + b[i] # (3) c = a + b if __name__ == '__main__': i = 0 # (1) while i < 500: invoke() i += 1 ``` Here is the trace of invoke visible in jitviewer (uploaded to https://postimg.org/image/kbntluw55/full/). Here are some questions I have that would really help me to get going: (1) Is there a better way to get loops hot? (2) I cannot really figure out what all those trace/loop parameters are. obviously i can guess some but most of them do not really make sense to me. Am I missing some settings to be able to display more information for the trace? In addition to that I do not really see any chance to generate a simd loop for (2). Every array access is (arguably) guarded by an array index overflow and I think to skip that check would be invalid. (3) There is no trace generated for this instruction? Does this internally call a c function? (4) What in our opinion makes sense to vectorize with simd instructions? Could you provide some sample loops/code (ranging from simple to complex loops)? I would really appreciate any help and/or push into the right direction. Best, Richard

5 10

Re: [pypy-dev] Vectorizing numpy traces
by Laura Creighton Feb. 24, 2015

Feb. 24, 2015

The PyPy core developers are at a sprint in Leysin right now, and they happen to be out for dinner right now. So you, alas, are likely to need to wait. Sorry for impatience .... Laura

1 0

freebsd 7 and 8 buildslave
by Maciej Fijalkowski Feb. 22, 2015

Feb. 22, 2015

there is a freebsd 8 buildslave called "ananke" which is offline, who is the admin? Also freebsd 7 fails to compile for an idiotic reason, should we take this one down? http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64/builds/290/st…

2 2

Fwd: pypy memory test
by Christian Walder Feb. 17, 2015

Feb. 17, 2015

Hi there, I seem to be running into a problem with freeing memory. Under OS X all is well, but under Linux there seems to be a problem in the case of linked data structures. I have attached a python program which demonstrates the problem. The output of the script for both linux and OS X are below. Thanks very much in advance, and for the amazing work on PyPy. OS X: 2.7.8 (857f34cd4254, Oct 14 2014, 22:01:17) [PyPy 2.5.0-alpha0 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] Darwin 14.1.0 ('', '', '') initial 20 MB 0 558 MB 0 30 MB 1 558 MB 1 31 MB 2 558 MB 2 32 MB Linux: 2.7.8 (dfffd5d7cc7e, Feb 08 2015, 14:59:48) [PyPy 2.5.0 with GCC 4.8.2] Linux 3.13.0-44-generic ('debian', 'jessie/sid', '') initial 82 MB 0 626 MB 0 626 MB 1 627 MB 1 627 MB 2 627 MB 2 627 MB

2 2

contribute in the project
by Ahmed Abdalrahman Feb. 17, 2015

Feb. 17, 2015

hello , my name is Ahmed , i want to join u and i am interested in open source what ever your will join google summer of code or not i want to coding , trying to learn and help you if i can please tell me, my experience , below average in python thanks for reading

2 1

GSoC 2015
by Rohan Goel Feb. 17, 2015

Feb. 17, 2015

Dear Developers, I am Rohan Goel , Computer Science undergraduate at BITS Pilani , India and am interested in working for your organization in GSoC 2015. As I am a beginner in open source coding , it would be great help if you guide me where to start from. My primary language of interest is Python. *Thanking You,* *Rohan Goel* *Under Graduate, B.E. (Hons) Computer Science Birla Institute of Technology and Science, Pilani* *K.K Birla Goa Campus* *+91 7722047225 **| rohangoel0296(a)gmail.com <rohangoel0296(a)gmail.com>*

2 1

Pyrlang Post
by 黄若尘 Feb. 17, 2015

Feb. 17, 2015

1 0

Re: [pypy-dev] report of some experience in Pyrlang and RPython
by Matti Picus Feb. 16, 2015

Feb. 16, 2015

Here's a quick edit job, again as both PDF and docx (with revision marks). Graphs are still missing axis labels and some kind of captioning Matti On Thu, Feb 12, 2015 at 7:23 AM, 黄若尘<hrc706(a)gmail.com> wrote: > I like the blog post. It's full of technical details and might need > some editing, but otherwise fine. I'm also happy to just send it as it > is (but please add numbers and explanations on axis of graphs > > On Thu, Feb 12, 2015 at 7:23 AM, 黄若尘<hrc706(a)gmail.com> wrote: > Hi, Fijal, > I have modified some contents of my post and added the section about the comparison among Pyrlang, Erlang and HiPE. > I think the last part of the comparison is interesting, which showed a slowing down when the scale of benchmark become quite huge. > The docx format seems has something wrong so I also attached the pdf one. > > Cheers, > Huang

3 2

could it be that cpyext is faster than python + cffi?
by Matti Picus Feb. 15, 2015

Feb. 15, 2015

I am close to releasing a blog post about numpy.linalg https://gist.github.com/mattip/25680e68fe7e2856fe0c Note the benchmark section, this is not a typo. Here's how I got to the conclusion that cpyext is faster than python+cffi. I installed numpy from the pypy/numpy repo on a nightly after 2.5.0 pip install git+https://bitbucket.org/pypy/numpy.git I then benchmarked the two pypy methods of calling numpy.linalg.inv() via this script which simply calls linalg.inv() for different sized ndarray. https://gist.github.com/mattip/a7b22dc237fb09a46758 $pypy inverse.py 10 UserWarning: npy_clear_floatstatus, npy_set_floatstatus_invalid not found warn('npy_clear_floatstatus, npy_set_floatstatus_invalid not found') after 999999, for n= 10, time=20.64, 20.64 msec/1000 loops and $USE_CPYEXT=yes pypy inverse.py 10 after 999999, for n= 10, time= 8.52, 8.52 msec/1000 loops Any thoughts? Comments on the blog post? Matti

3 5

mix-and-match approach to implementation decisions
by anatoly techtonik Feb. 15, 2015

Feb. 15, 2015

Hi, Reading http://rpython.readthedocs.org/en/latest/ to see if I can understand the text without a formal CS education. What does "mix-and-match approach to implementation decisions" mean? -- anatoly t.

2 3

report of some experience in Pyrlang and RPython
by 黄若尘 Feb. 13, 2015

Feb. 13, 2015

Hi, everyone, I tried to wrote down the experience I have done recently about my project of Pyrlang, but I’m not sure if there are enough explanation for details of JIT, and if there are some inappropriate expressions. So I think it may be better to show it to you then post it to the PyPy Blog. Please feel free to add or modify it or delete verbose expressions (if any). Thank you in advance. Ruochen Huang

3 5

numpy installation
by Maciej Fijalkowski Feb. 12, 2015

Feb. 12, 2015

Hey There is STILL no clear installation path for PyPy + numpy, with no instructions anywhere, can you please do something about it? Ideally pip install numpy-pypy should work and we should be done (this + info on pypy.org will do) I asked about this months ago, if noone picks it up, I'll deal with it, but this is by far the biggest problem for PyPy+numpy users (they can't install it). Cheers, fijal

7 8

Leysin Sprint location
by Armin Rigo Feb. 9, 2015

Feb. 9, 2015

Hi all, If you're coming back to Leysin for the sprint at the end of February: warning, the main entrance to the two chalets Ermina was moved from the upper chalet to the lower chalet. In case you need it, here's a map showing how to go to the lower chalet (red marker) when you're standing in front of the upper chalet: https://www.google.ch/maps/dir/46.3488973,7.0201613/46.3486126,7.0210687/@4… A bientôt, Armin.

1 0

Playing with PyPy and Django
by Tin Tvrtković Feb. 9, 2015

Feb. 9, 2015

Hello, PyPy folks! While trying to speed up one of my Django sites, I noticed a new version of PyPy had just been released. So I grabbed a fresh download of PyPy 3 (since this is a Python 3 codebase) and tried taking it out for a spin. However, as far as I can see, whatever I try PyPy is consistently slower than CPython for this. Since this is a proprietary site, I've basically ripped out all the code except my settings.py and my requirements; and am benchmarking the Django admin index. The results are about the same. I've set up a small repo that can be used to reproduce the environment: https://github.com/Tinche/PyPy-Django-Playground. There's additional info in the README there. These tests have been carried out on Ubuntu Trusty, 64-bit. CPython 3 is the system Python, 3.4. PyPy has been downloaded from the official site and unzipped. So what I basically do is set up an admin session, and use the Django main admin page. 200 warmup requests, then 100 benchmarked requests, look at the mean request time. Some results: Django's runserver, DEBUG mode: PyPy3 485.389 [ms] CPython 3.4 105.777 [ms] Django's runserver, no debug: PyPy3 44.661 [ms] CPython 3.4 18.697 [ms] Gunicorn, 1 worker, no debug: PyPy3 28.615 [ms] CPython 3.4 13.532 [ms] I don't exactly claim to be an expert on benchmarking, but assuming my site is similar to the Django admin, CPython's gonna be giving me better performance. Also the debug runserver performance is kinda worrying. Nobody's going to be running this in production, but it makes development a little slower and more annoying than it should be. Is there anything to make PyPy more competitive in these kinds of scenarios? Kind regards, Tin

5 16

Another idea for improving warm-up times
by Armin Rigo Feb. 7, 2015

Feb. 7, 2015

Hi all, Here's an idea that came up today on irc (thanks the_drow_) while discussing again saving the generated assembler into files and reloading it on the next run. As you know we categorize this idea as "can never work", but now I think that there is a variant that could work, unless I'm missing something. What *cannot* work is storing in the files any information about Python-level data, like "this piece of assembler assumes that there is a module X with a class Y with a method Z". I'm not going to repeat again the multiple problems with that (see http://pypy.readthedocs.org/en/latest/faq.html#couldn-t-the-jit-dump-and-re…). However, it might be possible to save enough lower-level information to avoid the problem completely. The idea would be that when we're about the enter tracing mode and there is a saved file, we use a "fast-path": * we find a *likely* candidate from the saved file, based on some explicitly-provided way to hash the Python code objects, for example (it doesn't have to be a guaranteed match) * this likely saved trace comes with a recording of all the *green* guards that we originally did (both promotions and things that just depend on purely green inputs). (This means extra work for making sure we save this information in the first place.) * we run it like in the blackhole interpreter, but checking the result of the green guards against the recorded ones * we also take all green values that we get this way and pass them as constants to the next step (see below, "**"). This means that we generalize (and lower-level-ize) the vague idea "a module X with a class Y with a method Z" to be instead a series of guards that need to give the same results as before. They would automatically check some subset of the new interpreter's state by comparing it against the old's --- but only as much as the actual loop happens to need. For example, if we had in the (old) normal trace a guard_value(p12, <constant pointer>), then of course it makes no sense to record the old interpreter's constant pointer, which will change. But it makes sense to record *what* was really deduced from this constant pointer, i.e. all the green getfields and getarrayitems we did. And for example, if it was a PyCode object, we would record the green switch that we did on the integer value that we got in the old interpreter (which is the next opcode), even though that's all green. That's the real condition: that we would follow the same path by constant-folding the decisions on the green variables. So, to finish the new interpreter's reloading: if the checking done above passes, the next step is a fast-path through the assembler. We "just" need to reload the saved assembler as a sequence of bytes, and fix all constants there. To continue the example above, if a piece of assembler was generated from the instruction guard_value(p12, <constant pointer>), then the saved file must contain enough information so that we know we must replace this old constant pointer's value in the assembler with the new constant pointer's value recorded above (at "**"). Overall, this would result in a much faster warm-up: faster tracing, and no optimization nor regular assembler generation at all --- only a very quick assembler reloading and fixing step. There are of course complications from the fact that we don't simply record loops, but bridges. They might be seen in a different order in the new process, so that when we are in the checking mode, we might run the start of the loop, but then jump into the bridge --- even though the loop was not fully seen so far. This is not impossible to implement by reloading the complete loop+bridge, but making the tail of the loop invalid until we really run into it (with an extra temporary guard). And I'm sure that unrolling will somehow come with its lot of funniness, as usual. However, does it sound reasonable at all, or am I missing something else? A bientôt, Armin.

2 2

Fwd: [pypy-commit] pypy default: Rename the --output option to --really-force-output, and document it
by Armin Rigo Feb. 6, 2015

Feb. 6, 2015

Hi Tobias, (answering an off-list-by-mistake mail) On 6 February 2015 at 13:50, Tobias Pape <Das.Linux(a)gmx.de> wrote: > So this is changed in rpython for a pypy issue? Yes, someone else made the same point on IRC. I have now reverted this change, and added a check to "targetpypystandalone.py" that you don't try to translate pypy with both "--shared" and "--output", which should be enough to avoid the original issue. A bientôt, Armin.

1 0

Re: [pypy-dev] [pypy-commit] pypy default: Rename the --output option to --really-force-output, and document it
by Maciej Fijalkowski Feb. 6, 2015

Feb. 6, 2015

This change breaks places that recommend you to use --output (e.g. check that you're not translating with the same executable) On Thu, Feb 5, 2015 at 5:52 PM, arigo <noreply(a)buildbot.pypy.org> wrote: > Author: Armin Rigo <arigo(a)tunes.org> > Branch: > Changeset: r75726:babfa1dd27a5 > Date: 2015-02-05 16:51 +0100 > http://bitbucket.org/pypy/pypy/changeset/babfa1dd27a5/ > > Log: Rename the --output option to --really-force-output, and document it > as "don't use with PyPy". > > diff --git a/rpython/config/translationoption.py b/rpython/config/translationoption.py > --- a/rpython/config/translationoption.py > +++ b/rpython/config/translationoption.py > @@ -151,7 +151,9 @@ > default=False, cmdline="--dont-write-c-files"), > ArbitraryOption("instrumentctl", "internal", > default=None), > - StrOption("output", "Output file name", cmdline="--output"), > + StrOption("output", "Output file name (don't change for PyPy!" > + " doesn't work with virtualenv+shared: issue 1971)", > + cmdline="--really-force-output"), > StrOption("secondaryentrypoints", > "Comma separated list of keys choosing secondary entrypoints", > cmdline="--entrypoints", default="main"), > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > https://mail.python.org/mailman/listinfo/pypy-commit

2 1

Faulty binary 2.5.0 package(s) on bitbucket download page
by Matti Picus Feb. 6, 2015

Feb. 6, 2015

1 1

A question about RPython's JIT in scheduler
by 黄若尘 Feb. 5, 2015

Feb. 5, 2015

Mr. Bolz, Hello, I’m a master student in Japan and this is the second time that I send a mail to you. :) Recently I’m implementing an Erlang interpreter by RPython, and I just have added a scheduler to my interpreter to simulate the multi-process in single-core. I compared two version of my interpreter, one has a scheduler, and one doesn’t, and I’m very surprised to find that there was only a very little overhead for the scheduler mechanism, in my benchmark it was only 3%. In my implementation, the scheduler has a run able queue, whose element is a tuple of an object which has a function for dispatch loop, a program counter and a reference to the Erlang byte code. While doing scheduling, the scheduler just dequeue an element from the run able queue, call the function for dispatch loop, the loop which only run for a limited time, and the scheduler will enqueue the element (tuple of object with dispatch loop, program counter and reference to the Erlang byte code) to the run able queue again. So in my opinion, it may be a trouble for JIT’s work because the dispatch loop is not always continuing, from the view of scheduler, the dispatch loop will run only a limited time of iterations, then be suspended, and resumed, and suspended again and so on. I think it may cause the JIT hard to do profiling, and I have no idea if the complied native code from JIT can be reuse when the dispatch loop resumed, either. From the benchmark I run I guess there may be some special cares taken to hold this situation, (actually I have also compared the JIT log generated for the two versions of interpreter below, it seemed quite similar at most of cases) so I’m just curious about how the JIT actually do under a scheduler? How does JIT overcome the trouble in this discontinuous environment? Best Regards, Ruochen Huang

2 1

Post on Swisscom ICT Blog "Python Performance: A Recommendation"
by Martin.Gfeller＠swisscom.com Feb. 5, 2015

Feb. 5, 2015

Dear all, I wrote a plug for PyPy on the Swisscom ICT Blog: http://ict.swisscom.ch/2015/02/pyperformance/? Kind regards Martin Gfeller

1 0

Re: [pypy-dev] Windows: pypyw.exe?
by Laura Creighton Feb. 4, 2015

Feb. 4, 2015

In a message of Wed, 04 Feb 2015 13:30:27 -0500, Yaacov Finkelman writes: >Hi, > >I was wondering what the plan is for fixing: >https://bitbucket.org/pypy/pypy/issue/1651/ > >It is just a hassle with each upgrade of pypy. > >I'd be happy to fix, but I don't know where to start. > >Best, > >Jacob Please excuse me for sending you a blank message. I was trying to test my 'send mail about bitbucket feature' and, well, i screwed it up and instead of sending me a private message about that you made a bitbucket issue mention .... it sent me that and you an empty string, null mail. very, very sorry, this will not happen again. I was careless. Laura Creighton

1 0