Mailman 3 January 2015 - pypy-dev

use as benchmark pypy vs python if you please
by Dima Tisnek 13 Sep '15

13 Sep '15

Attached is n-queens solver (pardon my naive algorithm), it runs: python 2.7.6: 17s pypy 2.4.0 alpha: 23s same nojit: 32s I've tried similar-looking algorithm for another problem before, and has similar results -- somehow pypy was slower. feel free to investigate / tweak or even use on speed.pypy.org

4 4

Poor performance for Krakatau
by Robert Grosse 10 Aug '15

10 Aug '15

In some cases, Pypy performs worse than CPython when running Krakatau, and it has gotten a lot worse from 2.3 to 2.5 My benchmark is https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (for future reference, commit d889c7f44723e6d66a3630681f0385f173317dc9) You can run it with pypy Krakatau\benchmark.py -path <path to Java rt.jar> I get the following timings CPython: 17.43s Pypy 2.3: 38.48s (build pypy-c-jit-71056-c8e3b8cbc843-win32) Pypy 2.4: 39.92s (build pypy-c-jit-72200-375133966c12-win32) Pypy 2.5: 49.94s (build pypy-c-jit-74404-9fd586fe0fe5-win32) Not only was it slower than CPython to begin with, but it got much worse recently.

3 4

A question about RPython's JIT in scheduler
by 黄若尘 05 Feb '15

05 Feb '15

Mr. Bolz, Hello, I’m a master student in Japan and this is the second time that I send a mail to you. :) Recently I’m implementing an Erlang interpreter by RPython, and I just have added a scheduler to my interpreter to simulate the multi-process in single-core. I compared two version of my interpreter, one has a scheduler, and one doesn’t, and I’m very surprised to find that there was only a very little overhead for the scheduler mechanism, in my benchmark it was only 3%. In my implementation, the scheduler has a run able queue, whose element is a tuple of an object which has a function for dispatch loop, a program counter and a reference to the Erlang byte code. While doing scheduling, the scheduler just dequeue an element from the run able queue, call the function for dispatch loop, the loop which only run for a limited time, and the scheduler will enqueue the element (tuple of object with dispatch loop, program counter and reference to the Erlang byte code) to the run able queue again. So in my opinion, it may be a trouble for JIT’s work because the dispatch loop is not always continuing, from the view of scheduler, the dispatch loop will run only a limited time of iterations, then be suspended, and resumed, and suspended again and so on. I think it may cause the JIT hard to do profiling, and I have no idea if the complied native code from JIT can be reuse when the dispatch loop resumed, either. From the benchmark I run I guess there may be some special cares taken to hold this situation, (actually I have also compared the JIT log generated for the two versions of interpreter below, it seemed quite similar at most of cases) so I’m just curious about how the JIT actually do under a scheduler? How does JIT overcome the trouble in this discontinuous environment? Best Regards, Ruochen Huang

2 1

PyPy improving generated machine code
by Richard Plangger 03 Feb '15

03 Feb '15

Hi, I'm a student at the technical university of Vienna and currently looking for a topic to complete my master thesis. I stumbled over PEP 484 that currently is discussed on the mailing list. It seems to me that this is going to become reality pretty soon. I had the idea that this additional type annotations could beneficial for JIT compilation. Two weeks ago someone already mentioned pretty much the same idea (https://mail.python.org/pipermail/pypy-dev/2015-January/013037.html). In this thread it was mentioned that to improve the compiled code more detailed information (such as an integer stays in range [0-50], ...) would be necessary to remove guards of a trace. I read the document "Tracing the Meta-Level: PyPy's Tracing JIT Compiler" that was published 2009 to understand the basics of how PyPy currently works. I assume that PyPy is still a trace JIT compiler. By using the PEP 484 proposal I think this opens up new possibilities. Using trace compilation as it is done in PyPy or SpiderMonkey makes a lot of sense because most of the time type information is not present prior the first execution. PEP 484 changes the game. After type inference has completed e.g. on a function it should not occur often that a variable's type is unknown. The document "Tracing the Meta-Level" already mentioned that when RPython is provided as input to PyPy it already infers the type. Is that true for not RPython programs as well? I think there are two possibilities to improve the generated machine code for PyPy: * Find a sensible sub set of optimizations that rely on the available type information and further improve the trace compilation * Evaluate other possibilities of inter procedural methods to compile good machine code or completely move to method based jit compilation. I could imagine to evaluate and implement this for my master thesis. What do you think? Would it benefit PyPy? Has anybody else started to implement something similar? Best, Richard

5 7

pickle in pypy is slow
by Gelin Yan 31 Jan '15

31 Jan '15

Hi All I noticed pickle in pypy is slower than cPickle in python (faster than pickle in python). Is it a feature? Regards gelin yan

3 2

pypy-c failing to find libpypy-c.so on freebsd
by Matti Picus 31 Jan '15

31 Jan '15

The freebsd builds are failing since we changed to --shared by default. While translation succeeds, the resulting pypy-c cannot find libpypy-c.so even though it seems to be copied properly. See for instance http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-9-x86-64/builds/462/st… It seems the $ORIGIN flag is somehow not functioning properly. I found this link http://stackoverflow.com/questions/6324131/rpath-origin-not-having-desired-… which would seems to suggest we need to add "-z origin" as well as -rpath=$ORIGIN to the linker flags. Could someone (Tobias is the admin of the buildbot, but anyone else is welcome to try) with a freebsd platform try to track this down? It should be enough to run python pytest.py rpython/translator/c/test/test_standalone.py -k shared --verbose -s on a pypy default repo after commit fa382e9b1c95, the test should fail. Then try to mess with the rpath_flags in rpython/translator/platform/posix.py till it passes Thanks Matti

2 1

starting 2.5 release cycle, help needed with macos
by Matti Picus 29 Jan '15

29 Jan '15

Buildbots for linux are green (arm and x86), windows seems as good as it gets. I have looked at the open issues, none seem like blockers. My personal baby, the ufuncapi branch, seems to be functioning after I found the "last bug" So I guess it is time to start the 2.5 release cycle, unless I missed something. We have a persistent crash with macos nightly builds in the _continuation module, help is needed to track it down http://buildbot.pypy.org/summary?builder=pypy-c-jit-macosx-x86-64 Armin suggested maybe it was shadowstack, I think I ruled that out by translating on x86 linux: https://gist.github.com/mattip/8407b7fa7dbe1cc2f786 Any help/criticism/comments are welcome Matti

3 2

rpython and pep 484
by Ho33e5 25 Jan '15

25 Jan '15

Hi everybody, firstly, this is just an email for personal interest and has nothing to do directly with development so this mailing list may not be quite the right place (I am going to hang around on #pypy...). I am a student and generally interested with the pypy development, especially with the rpython language, and I have some general questions: What is your view on the new typing/mypy things that are happening on python-dev (pep 484)? What I mean is will this make the typing system of rpython evolve? Could RTyper be adapted to work on pep 484 annotations (would it actually be useful)? I read a bit of the paper about rpython listed on the docs and i had the feeling that your typing is a bit more low level. The quite different goals and contraints that the 2 type system have may explain that they look different, but could there be an interaction (in one way or another)? An other question that is related: it's maybe early to think about that but could it be reasonable to expect that pypy will better optimize pep-484-annotated python programs? The thrusting of these user annotations is indeed a problem, so a pypy option could specify that we want it to thrust the type annotations. It may then be worth just writing programs in rpython directly. These questions are quite hypothetical so I don't expect concrete answers, just thoughts! If someone wants to react to this or point me to other (theoretical) ressources about rpython... :) Bonsoir, Peio

4 3

RFC: Copy-on-write list slices
by Mike Kaplinskiy 23 Jan '15

23 Jan '15

Hey folks, https://bitbucket.org/mikekap/pypy/commits/b774ae0be11b2012852a175f4bae4484… has an implementation of list slicing that copies the data on write. (The third idea from http://doc.pypy.org/en/latest/project-ideas.html .) It's a first pass (and also my first time working on the pypy codebase), so I wanted to solicit some feedback. I'm curious if this was even the right direction and if I'm actually breaking/slowing something down without realizing it. Also would anyone happen to know some representative stress/performance tests I could run? I ran some simple tests myself (some things got slightly faster), but I doubt that's enough :) Thanks, Mike. (Aside: there is a pull request @ https://bitbucket.org/pypy/pypy/pull-request/282/add-a-copy-on-write-slice-… for this commit, but I clearly messed something up with hg - the diff is from an earlier copy and bitbucket doesn't seem to want to pick it up.)

3 2

errno and GetLastError in RPython
by Armin Rigo 23 Jan '15

23 Jan '15

Hi all, I recently merged the "errno-again" branch. This branch moves the reading/saving of errno (and on Windows Get/SetLastError) closer to the actual function call. It should avoid bugs about rarely getting the wrong value for errno, in case "something special" happened: for example, it was not impossible that a call to malloc would invoke the GC at precisely the wrong spot, which might need to ask the OS for more memory, which would overwrite the current value of errno. The bug actually showed up on the Windows buildbots for GetLastError(), which would in some cases incorrectly return 0 just if the code happened to be JIT-traced (not before and not after). This is now fixed. It means any RPython project needs to be updated when it upgrades to the trunk version of the RPython translation toolchain. The fix is rather mechanical. Replace rposix.get_errno() with rposix.get_saved_errno(). Importantly, review each place that you change. You need to make sure which external function call is done before (usually in the few lines before). Once you're sure which function's errno is being checked, go to the declaration of that function, which should be using rffi.llexternal(). Add the keyword argument "save_err=rffi.RFFI_SAVE_ERRNO". This forces errno to be saved immediately after the function call, into the so-called "saved errno". This "saved errno" is another thread-local variable, which rposix.get_saved_errno() returns. Similarly with rwin32.GetLastValue() -> rwin32.GetLastValue_saved() + rffi.RFFI_SAVE_LASTERROR. If there are cases with rposix.set_errno(0), they can be killed and the following function given the flag "RFFI_ZERO_ERRNO_BEFORE". See the new docstrings of the rposix.get/set_saved_errno() and rwin32.Get/SetLastValue_saved() for more details. A bientôt, Armin.

1 0