Mailman 3 January 2013 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus 22 Dec '13

22 Dec '13

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4

Parallella open hardware platform
by John Camara 07 Feb '13

07 Feb '13

A couple of days ago I heard about the Parallella [1] project which is an open hardware platform similar to the Raspberry Pi but with much higher capabilities. It has a Zynq Z-7010 which has both a dual core ARM A9 (800 MHz) processor and a Artix-7 FPGA, a 16 core Epiphany multicore accelerator, 1GB ram (see [2] for more info) and currently boots up in Ubuntu. The goal of the Parallella project is to develop an open parallel hardware platform and development tools. Recently they announced support for Python with Mark Dewing [3] leading the effort. I had asked Mark if he considered PyPy but at this time he doesn't have time for this investigation and he reposted my comment on the forum [4] with a couple of question. Maybe one of you could answer them. Working with the Parallella project maybe a good opportunity for the PyPy project from both a PR perspective and as well as the technical challenges it would present. On the technical side it would give the opportunity to test STM on a reasonable number of cores while also dealing with cores from different architectures (ARM and Epiphany). I could see all the JITting occurring on the ARM cores with it producing output for both architectures based on which type of core STM decides to use for a chunk of work to execute on. Of course there is also the challenge of bridging between the 2 architectures. Maybe even some of the more expensive STM operations could be offloaded to the FPGA or even a limited amount of very hot sections of code could be JITted to the FPGA (although this might be more work than its worth). >From a PR perspective PyPy needs to excel at some niche market so that the PyPy platform can take off. When PyPy started concentrating on the scientific market with increasing support for Numpy I thought this would be the niche market that would get PyPy to take off. But there have been a couple of issue with this approach. There is a tremendous amount of work that needs to be done so that PyPy can look attractive to this niche market. It requires supporting both NumPy and SciPy and their was an expectation that if PyPy supports NumPy others would come to help out with the SciPy support. The problem is that there doesn't seam to be many who are eager to pitch in for the SciPy effort and there also has not been a whole lot willing to help will the ongoing NumPy work. I think in general the ratio of people who use NumPy and SciPy to those willing to contribute is quite small. So the idea of going after this market was a good idea and can definitely have the opportunity to showing the strength of PyPy project it hasn't done much to improve the image of the PyPy project. It also doesn't help that there is some commercial interests that have popped up recently that have decided to play hard ball against PyPy by spreading FUD. Unlike the Raspberry Pi hardware which can only support hobbyist the Parallella hardware can support both hobbyists and commercial interests. They cost $100 which is more than the $35 for Raspberry Pi but still within reach of most hobbyists and they didn't cut out the many features that are needed for commercial interests. The Parallella project raised nearly $0.9 million on kickstarter [5] for the project with nearly 5000 backers. Since many who will use the Parallella hardware also have experience on embedded systems they and are more likely used to writing low level code in assembly, FPGAs, and even lots of C code and I'm sure have hit many issues with programming in parallel/multithreaded and would welcome a better developer experience. I bet many of them would be willing to contribute both financially and time to supporting such an effort. I believe the Architecture of PyPy could lend it self to becoming the core of such a development system and would allow Python to be used in this space. This could provide a lot of good PR for the PyPy project. Now I'm not saying PyPy shouldn't devote any more time to supporting NumPy as I'm sure when PyPy has very good support for both NumPy and SciPy it's going to be a very good day for all Python supporters. I just think that the PyPy team needs to think about a strategy that in the end will help its PR and gain support from a much larger community. This project is doing a lot of good things technically and now it just needs to get the attention of the development community at large. Now I can't predict if working with the Parallella project would be the break though in PR that PyPy needs but it's at least an option that's out there. BTW I don't have any commercial interests in the Parallella project. If some time in the future I use their hardware it would likely be as a hobbyist and it would be nice to program it in Python. My real objective of this post to see the PyPy project gain wider interest as it would be a good thing for Python. [1] - http://www.parallella.org/ [2] - http://www.parallella.org/board/ [3] - http://forums.parallella.org/memberlist.php?mode=viewprofile&u=3344 [4] - http://forums.parallella.org/viewtopic.php?f=26&t=139 [5] - http://www.kickstarter.com/projects/adapteva/parallella-a-supercomputer-for… John

4 9

(unexpectedly) bad performance of stackless and greenlet on PyPy
by Denis Bilenko 31 Jan '13

31 Jan '13

Hi! I've micro-benchmarked stackless and greenlet module in PyPy and was quite surprised with the results. According to this benchmark stackless.channel is 100x times slower on PyPy (vs Stackless) and greenlet is 20x times slower on PyPy (vs CPython). https://gist.github.com/4412582 Is this a bug in benchmark (or PyPy) or is it the expected current state of stackless features on PyPy? Cheers, Denis.

2 2

Translating safely (without executing)
by Kirk Liberty 30 Jan '13

30 Jan '13

Hello all, Is it possible to use PyPy's translation tool without having it execute the program? I think this would be important for programs with infinite loops, or ones which could do potentially dangerous things ie. when trying to translate a program to control a GPU fan, the fan was commanded off during a high work load. Thank you, Kirk

5 10

Extending include path when translating?
by Skip Montanaro 27 Jan '13

27 Jan '13

I'm back messing around, trying to get PyPy going on Solaris. I worked around one issue to get it to use the -std=c99 flag with GCC. I stumbled on a bug in ctypes/util.py and found a patch which solves that problem: http://bugs.python.org/issue5289 My latest issue is slightly deeper in translation: [translation:WARNING] The module '_hashlib' is disabled [translation:WARNING] because importing pypy.module._ssl.interp_ssl raised CompilationError [translation:WARNING] CompilationError(out=""" [translation:WARNING] CompilationError(err=""" [translation:WARNING] platcheck_6.c:29:25: error: openssl/ssl.h: No such file or directory [translation:WARNING] platcheck_6.c:30:25: error: openssl/err.h: No such file or directory [translation:WARNING] platcheck_6.c:31:26: error: openssl/rand.h: No such file or directory [translation:WARNING] platcheck_6.c:32:25: error: openssl/evp.h: No such file or directory [translation:WARNING] platcheck_6.c:33:30: error: openssl/ossl_typ.h: No such file or directory [translation:WARNING] platcheck_6.c:34:28: error: openssl/x509v3.h: No such file or directory [translation:WARNING] """)""") [translation:WARNING] The module 'pyexpat' is disabled [translation:WARNING] because importing pypy.module.pyexpat.interp_pyexpat raised CompilationError [translation:WARNING] CompilationError(out=""" [translation:WARNING] CompilationError(err=""" [translation:WARNING] platcheck_7.c:29:19: error: expat.h: No such file or directory [translation:WARNING] """)""") [translation:WARNING] The module '_ssl' is disabled [translation:WARNING] because importing pypy.module._ssl.interp_ssl raised CompilationError [translation:WARNING] CompilationError(out=""" [translation:WARNING] CompilationError(err=""" [translation:WARNING] platcheck_14.c:29:25: error: openssl/ssl.h: No such file or directory [translation:WARNING] platcheck_14.c:30:25: error: openssl/err.h: No such file or directory [translation:WARNING] platcheck_14.c:31:26: error: openssl/rand.h: No such file or directory [translation:WARNING] platcheck_14.c:32:25: error: openssl/evp.h: No such file or directory [translation:WARNING] platcheck_14.c:33:30: error: openssl/ossl_typ.h: No such file or directory [translation:WARNING] platcheck_14.c:34:28: error: openssl/x509v3.h: No such file or directory [translation:WARNING] """)""") I actually have OpenSSL and Expat available, but their headers aren't in standard locations like /usr/include. Is there a way to extend the include search path (and probably library search path) so the translator can find them in their non-standard locations? Thanks, Skip Montanaro

2 1

FreeBSD 9-STABLE
by exarkun＠twistedmatrix.com 24 Jan '13

24 Jan '13

Hello all, Looking at PyPy's buildbot, it appears that the current FreeBSD builder is a FreeBSD 7 system. I'm curious who out there is interested in FreeBSD 7 in particular, and whether anyone is interested in newer versions (say, FreeBSD 9, recently released). If there's interest, I think I can offer a FreeBSD 9 builder, either to replace or augment the current FreeBSD builder. Jean-Paul

2 1

python bytecode optimization.
by Robert Zaremba 21 Jan '13

21 Jan '13

Hi, I really like the work you do and try to support by trying to make all my python project runing under PyPy, being PyPy proponent, convince others to PyPy, putting some founding, talk about PyPy. I try to learn PyPy architecture and someday contribute (if I got more free time). I know that PyPy bind a lot of optimizations. My question is how it looks from bytecode interpretation (so without JITing)? I learned about AST optimizers for CPython (like https://bitbucket.org/haypo/astoptimizer, hotpy). Does PyPy bytcode interpreter implements some of them on bytecode level? I know that PyPy produce similar bytecode as CPython (http://doc.pypy.org/en/latest/architecture.html#the-python-interpreter), so I guess PyPy doesn't introduce such optymalizations. But maybe on some other interpreter level? PyPy fail on some astoptimizer tests. It is pure Python, but manipulates AST. I make a little introspection and i think, the reason is due to small bytecode differences? The AST (resulted and expected) looks similar, by the PyPy one is simpler.

4 4

RPython pypy split merge
by Maciej Fijalkowski 20 Jan '13

20 Jan '13

We are in the progress of splitting the RPython Toolchain from PyPy and merged the split-rpython branch into default. Existing code needs to be changed, usually just import renames. Most importantly: pypy.annotation -> rpython.annotator pypy.objspace.flow -> rpython.flowspace pypy.jit -> rpython.jit pypy.rlib -> rpython.rlib pypy.rpython -> rpython.rtyper pypy.translator -> rpython.translator Other changes: - py.py is now called pyinteractive.py. - pypy.module.signal.interp_signal is now rpython.rlib.rsignal - autopath.py has been killed I hope I didn't break your code too much! - Alexander Hesse (Aquana)

1 0

NumPyPy 2013 Developer Position
by Maciej Fijalkowski 16 Jan '13

16 Jan '13

Introduction ------------ Proposed herein is a part-time fellowship for developing NumPy in PyPy. The work will initially consist of 100 hours with the possibility of extension, until the funds run out. Development and improvement of PyPy's NumPyPy (as with most Open Source and Free Software) is done as a collaborative process between volunteer, paid, and academic contributors. Due to a successful funding drive but a lack of contributors willing to work directly for PyPy, we find ourselves in the enviable situation of being able to offer this position. Background ---------- PyPy's developers make all PyPy software available to the public without charge, under PyPy's Open Source copyright license, the permissive MIT License. PyPy's license assures that PyPy is equally available to everyone freely on terms that allow both non-commercial and commercial activity. This license allows for academics, for-profit software developers, volunteers and enthusiasts alike to collaborate together to make a better Python implementation for everyone. NumPy support for PyPy is licensed similarly, and therefore NumPy in PyPy support can directly help researchers and developers who seek to do numeric computing but want an easier programming language to use than Fortan or C, which is typically used for these applications. Being licensed freely to the general public means that opportunities to use, improve and learn about how NumPy in PyPy works itself will be generally available to everyone. The Need for a Part-Time Developer ----------------------------------- NumPy project in PyPy has seen some slow, but steady progress since we started working about a year ago. On one hand, it's actually impressive what we could deliver with the effort undertaken, on the other hand, we would like to see the development accelerated. PyPy has strict coding, testing, documentation, and review standards, which ensures excellent code quality, continually improving documentation and code test coverage, and minimal regressions. A part-time developer will be able to bring us closer to the goal of full numpy-api implementation and speed improvements. Work Plan --------- The current proposal is split into two parts: * **Compatibility**: This part covers the core NumPy Python API. We'll implement most NumPy APIs that are officially documented and we'll pass most of NumPy's tests that cover documented APIs and are not implementation details. Specifically, we don't plan to: * implement NumPy's C API * implement other scientific libraries, like SciPy, matplotlib or biopython * implement details that are otherwise agreed by consensus to not have a place in PyPy's implementation of NumPy or agreed with NumPy community to be implementation details * **Speed**: This part will cover significant speed improvements in the JIT that would make numeric computations faster. This includes, but is not necesarilly limited to: * write a set of benchmarks covering various use cases * teaching the JIT backend (or multiple backends) how to deal with vector operations, like SSE * experiments with automatic parallelization using multiple threads, akin to numexpr * improving the JIT register allocator that will make a difference, especially for tight loops As with all speed improvements, it's relatively hard to predict exactly how it'll cope, however we expect the results to be withing an order of magnitude of handwritten C equivalent. Position Candidate -------------------- We would like people who are proficient in NumPy and PyPy (but don't have to be core developers of either) to step up. The developer selection will be done by consensus of PyPy core developers and consulted with the Software Freedom Conservancy for lack of conflict of interest. The main criterium will be past contributions to the PyPy project, but they don't have to be significant in size. A candidate for the Developer position will demonstrate the following: * The ability to write clear, stable, suitable and tested code * The ability to understand and extend the JIT capabilities used in NumPyPy. * A positive presence in PyPy's online community on IRC and the mailing list. Ideally the Developer will also: * Have familiarity with the infrastructure of the PyPy project (including bug tracker and buildbot). * Have Worked to provide education or outreach on PyPy in other forums such as workshops, conferences, and user groups. Conservancy and PyPy are excited to announce the Developer Position. Renumeration for the position will be at the rate of 60 USD per hour, through the Software Freedom Conservancy. PyPy community is promising to provide necessary guidance and help into the current codebase, however we expect a successful candidate to be able to review code and incorporate external patches within two months of the starting date of the contract. Candidates should submit their proposal (including their CV) to: `pypy-z(a)python.org`_ The deadline for this initial round of proposals is February 1, 2013. .. _`pypy-z(a)python.org`: mailto:pypy-z@python.org Cheers, fijal and the PyPy team

1 0