Mailman 3 March 2015 - pypy-dev

use as benchmark pypy vs python if you please
by Dima Tisnek 13 Sep '15

13 Sep '15

Attached is n-queens solver (pardon my naive algorithm), it runs: python 2.7.6: 17s pypy 2.4.0 alpha: 23s same nojit: 32s I've tried similar-looking algorithm for another problem before, and has similar results -- somehow pypy was slower. feel free to investigate / tweak or even use on speed.pypy.org

4 4

Poor performance for Krakatau
by Robert Grosse 10 Aug '15

10 Aug '15

In some cases, Pypy performs worse than CPython when running Krakatau, and it has gotten a lot worse from 2.3 to 2.5 My benchmark is https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (for future reference, commit d889c7f44723e6d66a3630681f0385f173317dc9) You can run it with pypy Krakatau\benchmark.py -path <path to Java rt.jar> I get the following timings CPython: 17.43s Pypy 2.3: 38.48s (build pypy-c-jit-71056-c8e3b8cbc843-win32) Pypy 2.4: 39.92s (build pypy-c-jit-72200-375133966c12-win32) Pypy 2.5: 49.94s (build pypy-c-jit-74404-9fd586fe0fe5-win32) Not only was it slower than CPython to begin with, but it got much worse recently.

3 4

Windows: pypyw.exe?
by Yaacov Finkelman 22 May '15

22 May '15

Hi, I was wondering what the plan is for fixing: https://bitbucket.org/pypy/pypy/issue/1651/ It is just a hassle with each upgrade of pypy. I'd be happy to fix, but I don't know where to start. Best, Jacob

4 8

PyPy 2.5.1 is live, please help get the word out
by Matti Picus 30 Mar '15

30 Mar '15

2 1

Pypy - Unable to find vcvarsall.bat
by Heng Cheong 30 Mar '15

30 Mar '15

I was reluctant to send this email but after more than a couple of thoughts, I decided what the heck and go for it.I am writing in relation to installing pypy.I was successful in installing numpy, pandas, and matplotlib, but not pypy.My problem is when I install it, I got this error "Unable to find vcvarsall.bat" statement.A simple Google found this problem to be widespread since many years ago, and I guess until today it is still unresolved.How come the pypy development community does not do anything about this?

3 3

CfP ICOOOLPS Workshop: Implementation Compilation Optimization of Object Oriented Languages Programming and Systems
by Carl Friedrich Bolz 30 Mar '15

30 Mar '15

CALL FOR PAPERS (DEADLINE EXTENSION) 10th Workshop on Implementation, Compilation, Optimization of Object-Oriented Languages, Programs and Systems ! (Collocated with Ecoop’15) Prague, Czech Republic, July 6th, 2015 [The deadline has been extended to April, 13th 2015 ] The ICOOOLPS workshop series brings together researchers and practitioners working in the field of OO languages implementation and optimization. ICOOOLPS key goal is to identify current and emerging issues relating to the efficient implementation, compilation and optimization of such languages, and outlining future challenges and research directions. CALL FOR PAPERS Topics of interest for ICOOOLPS include, but are not limited to: - implementation of fundamental OO and OO-like features (e.g. inheritance, parametric types, memory management, objects, prototypes), - runtime systems (e.g. compilers, linkers, virtual machines, garbage collectors), - optimizations (e.g. static or dynamic analyses, adaptive virtual machines), - resource constraints (e.g. time for real-time systems, space or low-power for embedded systems) and relevant choices and tradeoffs (e.g. constant time vs. non-constant time mechanisms, - separate compilation vs. global compilation, - dynamic loading vs. global linking, dynamic checking vs. proof-carrying code…). SUBMISSION TYPES ICOOOLPS is not a mini-conference; it is a workshop designed to facilitate discussion and the exchange of ideas between peers. ICOOOLPS therefore welcomes both position (1—4 pages) and research (max. 10 pages) papers. Position papers should outline interesting or unconventional ideas, which need not be fully fleshed out. Research papers are expected to contain more complete ideas, but these need not necessarily be fully complete as with a traditional conference. Authors will be given the option to publish their papers (short or long) in the ACM Digital Library if they wish. Submissions must be written in English, formatted according to ACM SIGPLAN Proceedings style. Please submit via the EasyChair submission site https://easychair.org/conferences/?conf=icooolps15. ICOOOLPS PROGRAM COMMITTEE Chair: Floréal Morandat (LaBRI/France) - Olivier Zendra (Inria, Loria/France) Carl Friedrich Bolz (King's College London) Eric Jul (Alcatel-Lucent Bell Labs) Tobias Pape (Hasso-Plattner-Institute, Potsdam) Jean Privat (Université du Québec à Montréal) Jeremy Singer (University of Glasgow) Gaël Thomas (Telecom SudParis) Laurence Tratt (King's College London) Jan Vitek (Northeastern University) Mario Wolczko (Oracle Labs) Carl Friedrich Bolz

1 0

vmprof compression
by John Camara 28 Mar '15

28 Mar '15

Hi Fijal, To recap and continue the discussion from irc. We already discussed that the stack id are based on a counter which is good but I also want to confirm that the ids have locality associated with the code. That is similar areas of the code will have similar ids. Just to make sure are not random with respect to the code otherwise compression will not be helpful. If the ids are random that would need to be corrected first. Right now the stack traces are written to the file repeating the following sequence MARKER_STACKTRACE count depth stack stack ... stack In order to get a high compression ratio it would be better to combine multiple stacktraces and rearrange the data as follows MARKER_COMPRESSED_STACKTRACES counts_compressed_length counts_compressed depths_compressed_length depths_compressed stacks_compressed_length stacks_compressed In order to build the compress data you will want to 3 pairs of 2 buffers. A pair of buffers for counts, depths, and stacks. Your profiller would be writing to one set of buffers and another thread would be responsible for compressing buffers that are full and writing them to the file. Once a set of buffers are full the profiller would start filling up the other set of buffers. For each set of buffers you need a variable to hold the previous count, depth, and stack id. They will be initialized to 0 before any data is written to an empty buffer. In stead of writing the actual count value into the counts buffer you will write the difference between the current count and the previous count. The reason for doing this is that the delta values will mostly be around 0 which will significantly improve the compression ratio without adding much overhead. Of course you would do the same for depths and stack ids. When you compress the data you compress each buffer individually to make sure like data is being compressed. Like data compresses better the unlike data and by saving deltas very few bits will be required to represent the data and you are likely to have long strings of 0s and 1s. I'm sure now you can see why I don't want stack ids being random. As if they are random then the deltas will be all over the place so you wont end up with long strings of 0s and 1s and random data itself does not compress. To test this out I wouldn't bother modifying the c code but instead try it out in Python to first make sure the compression is providing huge gains and figure out how to tune the algorithm without having to mess with the signal handlers and writing the code for the separate thread and dealing issues such as making sure you don't start writing to a buffer before the thread finished writing the data to the file, etc. I would just read an existing profile file and rewrite it to a different file by rearranging the data and compressing the delta as I described. You can get away with one set of buffers as you wouldn't be profiling at the same time. To tune this process you will need to determine the appropriate number of stack traces that is small enough to keep memory down but large enough so that the overhead associated with compression small. Maybe start of with about 8000 stack traces. I would try gzip, bz2, and lzma and look at their compression ratios and times. Gzip is general faster than bz2 and lzma is the slowest. On the other hand lzma provides the best compression and gzip the worse. Since you will be compressing deltas you most likely can get away with using the fastest compression options under each compressor and not effect the compression ratio. But I would test it to verify this as it does depend on the data being compressed whether or not this is true. Also one option that is available in lzma is the ability to set the width of the data to look at when looking for patterns. Since you are saving 32 or 64 bit ints depending on the platform you can set the option to either 4 or 8 bytes based on the platform. I don't believe qzip or bz2 have this option. By setting this option in lzma you will likely improve the compression ratio. You may find that counts and depths give similar compression, between the 3 compression types in which case just use the fastest which will likely be gzip. On the other hand maybe the stack ids will be better off using lzma. This is also another reason to separate out, like data, as it gives you an option to use the fastest compressors for some data types while using others to provide for better compression. I would not be surprised if this approach achieves a compression ratio better than 100x but that will be heavily dependent on how local the stack ids are. Also don't forget about simple things like not using 64 bit ints when you can get away with smaller ones. Also for a slight variation to the above. If you find most of your deltas are < 127 you could write them out as 1 byte and when greater than 127 write them out as a 4 byte int with the high bit set. If you do this then don't set the lzma option to 4 or 8 byte boundaries as now your data is a mixture of 1 and 4 byte values. This sometimes can provide huge reductions in compression times without much effect on the overall compression ratio. Hope you find this helpful. John

4 4

Mod typing error
by Timothy Baldridge 28 Mar '15

28 Mar '15

I have some RPython that does this: return a % b while in another function I'm calling the same, but the two functions differ in the types of a and b (mix of ints and floats). However, during translation I'm getting a blocked block exception. translation:ERROR] AnnotatorError: [translation:ERROR] [translation:ERROR] Blocked block -- operation cannot succeed [translation:ERROR] [translation:ERROR] v12 = mod(v10, v11) [translation:ERROR] [translation:ERROR] In <FunctionGraph of (pixie.vm.numbers:1)_rem_Float_Integer at 0x105b80490>: [translation:ERROR] no source! [translation:ERROR] Known variable annotations: [translation:ERROR] v10 = SomeFloat() [translation:ERROR] v11 = SomeInteger(knowntype=int, nonneg=False, unsigned=False) [translation:ERROR] [translation:ERROR] Blocked block -- operation cannot succeed [translation:ERROR] [translation:ERROR] v15 = mod(v13, v14) [translation:ERROR] [translation:ERROR] In <FunctionGraph of (pixie.vm.numbers:1)_rem_Integer_Float at 0x1059ac550>: [translation:ERROR] no source! [translation:ERROR] Known variable annotations: [translation:ERROR] v13 = SomeInteger(knowntype=int, nonneg=False, unsigned=False) [translation:ERROR] v14 = SomeFloat() [translation:ERROR] [translation:ERROR] Blocked block -- operation cannot succeed [translation:ERROR] [translation:ERROR] v18 = mod(v16, v17) [translation:ERROR] [translation:ERROR] In <FunctionGraph of (pixie.vm.numbers:1)_rem_Float_Float at 0x10544a650>: [translation:ERROR] no source! [translation:ERROR] Known variable annotations: [translation:ERROR] v16 = SomeFloat() [translation:ERROR] v17 = SomeFloat() [translation:ERROR] Do we need another entry in rtyper? Looking at rtyper/rfloat.py I see entries on how to type add, sub, etc, but nothing for mod. Thanks, Timothy

3 2

Release 2.5.1 is close, please help test
by Matti Picus 25 Mar '15

25 Mar '15

The 2.5.1 release is almost ready. The release notice is here http://doc.pypy.org/en/release-2.5.x/release-2.5.1.html Note it has a broken link, the whatsnew is here instead http://doc.pypy.org/en/release-2.5.x/whatsnew-2.5.1.html This will be fixed when I merge the release branch back into default together with the official release The files ready for repackaging are here http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e… http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e… http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e… http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e… http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e… http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e… http://buildbot.pypy.org/nightly/release-2.5.x/pypy-c-jit-76372-8e24dac0b8e… https://bitbucket.org/pypy/pypy/get/release-2.5.1.zip https://bitbucket.org/pypy/pypy/get/release-2.5.1.tar.gz Could people please take a few minutes to test them out on their favorite platform? Also, any corrections, additions, or edits to the release notice would be welcome, especially a name instead of XXXXXX Perhaps from http://xkcd.com, something to do with bromeliads http://www.bsi.org/brom_info/what.html, maybe we can draw a parallel between the spread of them and the spreading popularity of rpython and cffi: "Within the last hundred years, bromeliads have become more widely used as ornamental plants. Originally only found in royal botanical gardens or the private greenhouses of wealthy Europeans, their popularity has spread to the masses. Today bromeliads are more available to the enthusiast than ever before. New species are still being discovered and plant breeders are developing ever more stunning hybrids to choose from." Matti

5 6

Custom types for annotating a flow object space
by Henry Gomersall 25 Mar '15

25 Mar '15

I'm looking at using PyPy's flow object space for an experimental converter for MyHDL (http://www.myhdl.org/), a Python library for representing HDL (i.e. hardware) models. By conversion, I mean converting the MyHDL model that represents the hardware into either Verilog or VHDL that downstream tools will support. Currently, there is a converter that works very well in many situations, but there are substantial limitations on the code that can be converted (much greater restrictions than RPython imposes), as well as somewhat frustrating corner cases. It strikes me that much of the heavy lifting of the conversion problem can be handled by the PyPy stack. My question then regards the following. MyHDL represents certain low level structures as python objects. For example, there is a notion of a signal, represented by a Signal object, that has a one to one mapping to the target HDL language. All the attributes of the Signal object describe how it should be converted. So, during annotation, the Signal object should be maintained as a base type, rather than burying deeper into the object to try and infer more about its type (which invariably breaks things due to RPython non-conformity). There are probably a few other types (though not many) that should be handled similarly. How does one instruct the translator to do this? Is it a case of writing a custom TranslationDriver to handle the custom types? Thanks for any help, Henry

5 14