Mailman 3 November 2013 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus 22 Dec '13

22 Dec '13

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4

FreeBSD 9.2 buildslave: first run
by Tobias Oberstein 16 Dec '13

16 Dec '13

Hi, the new FreeBSD buildslave has run the first time: http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-9-x86-64/builds/6 PyPy was built successfully, but there are a couple of issues: https://bugs.pypy.org/issue1637 https://bugs.pypy.org/issue1638 https://bugs.pypy.org/issue1639 I'll address those. There is another issue that bugs me .. look at the built times: [Timer] Timings: [Timer] annotate --- 576.0 s [Timer] rtype_lltype --- 1323.7 s [Timer] pyjitpl_lltype --- 1098.6 s [Timer] backendopt_lltype --- 314.2 s [Timer] stackcheckinsertion_lltype --- 161.5 s [Timer] database_c --- 451.3 s [Timer] source_c --- 668.5 s [Timer] compile_c --- 2403.8 s [Timer] =========================================== [Timer] Total: --- 6997.6 s and compare with "Linux 64 on allegro64": [Timer] Timings: [Timer] annotate --- 641.5 s [Timer] rtype_lltype --- 1528.5 s [Timer] pyjitpl_lltype --- 938.7 s [Timer] backendopt_lltype --- 242.2 s [Timer] stackcheckinsertion_lltype --- 206.8 s [Timer] database_c --- 325.3 s [Timer] source_c --- 357.7 s [Timer] compile_c --- 246.4 s [Timer] =========================================== [Timer] Total: --- 4487.1 s ***** Why the heck does "compile_c" take 10x the time on FreeBSD? Note that FreeBSD builder uses Clang .. could that be a reason? Any hints on that are welcome! Cheers, Tobias

3 3

Pluggable HTM
by Dimitri Vorona 09 Dec '13

09 Dec '13

Hi, the original STM proposal spoke of HTM as of a thing of a far future. Now, Haswells are out and provide built-in HTM support in form of TSX. In the near future I expect more and more systems to have it. Are there plan to make PyPy use HTM if it is available on the system? Regards, Dimitri.

3 2

clarification cffi vs _ffi_ vs _ffi vs libffi vs _rawffi vs rffi
by Dima Tisnek 03 Dec '13

03 Dec '13

there seems to be some confusion (could be all mine) between N terms related to FFI: are there both _ffi_ and _ffi, or is one of those a typo? extending page mentions libffi and not cffi, is it out of date, or is there a reason to stay away from cffi, although recommended in general faq page? in my own testing, I noticed that equivalent code cffi code is faster than ctypes, is there an explanation for this? thanks, d. docs in question: http://doc.pypy.org/en/latest/faq.html http://doc.pypy.org/en/latest/extending.html http://doc.pypy.org/en/latest/rffi.html

4 6

PyPy3 release?
by Armin Rigo 03 Dec '13

03 Dec '13

Hi all, Any volunteer to handle the PyPy3 release corresponding to PyPy 2.2? A bientôt, Armin.

5 6

Proposal for sprint in Tallinn, Estonia
by Ahti Heinla 02 Dec '13

02 Dec '13

Hi, I am new to PyPy, but very impressed with what you guys have done. Myself I am best known for having been a founding engineer and Chief Technical Architect for Skype. Python is my favourite language, and I want to help PyPy somehow. How about I organise a sprint in Tallinn, Estonia (where I live)? I can hopefully drum up some interest among local developers, perhaps also get some top ex-Skype engineers to join. The requirement to contribute a week full-time is a deterrent for many though, so I am not sure yet how many people I can get. Myself I have no background in interpreters/compilers, but have written code for 32 years, often optimised lowlevel code (assembly, C++). I would need some hand-holding to get started, but I am sure I'd be productive in a day or two. Ahti ahtih on IRC

4 5

Leysin sprint (11-19 January 2014)
by Armin Rigo 30 Nov '13

30 Nov '13

===================================================================== PyPy Leysin Winter Sprint (11-19st January 2014) ===================================================================== The next PyPy sprint will be in Leysin, Switzerland, for the ninth time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. ------------------------------ Goals and topics of the sprint ------------------------------ * Py3k: work towards supporting Python 3 in PyPy * NumPyPy: work towards supporting the numpy module in PyPy * STM: work towards supporting Software Transactional Memory * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski. ----------- Exact times ----------- For a change, and as an attempt to simplify things, I specified the dates as 11-19 January 2014, where 11 and 19 are travel days. We will work full days between the 12 and the 18. You are of course allowed to show up for a part of that time only, too. ----------------------- Location & Accomodation ----------------------- Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connexion with wireless installed. You can of course arrange your own lodging anywhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-) Please *confirm* that you are coming so that we can adjust the reservations as appropriate. The rate so far has been around 60 CHF a night all included in 2-person rooms, with breakfast. There are larger rooms too (less expensive per person) and maybe the possibility to get a single room if you really want to. Please register by Mercurial:: https://bitbucket.org/pypy/extradoc/ https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2… or on the pypy-dev mailing list if you do not yet have check-in rights: http://mail.python.org/mailman/listinfo/pypy-dev You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around -- bring a EU-format power strip if you have one. Armin Rigo

1 0

can we get stdlib-2.7.5 merged?
by Matti Picus 29 Nov '13

29 Nov '13

There are only a handful (ok - 10, two others are from the nightly default) of failures on the stdlib-2.7.5 branch: http://buildbot.pypy.org/summary?branch=stdlib-2.7.5 It seems the major one is handling eintr signal on file reading, which AFAICT would fix more or less 3 of the failures. The others seem to be unrelated to one-another. Would anyone like to take on some of this work? Matti

4 3