Mailman 3 September 2013 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus 22 Dec '13

22 Dec '13

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4

lang-scheme development
by Weikun Yang 25 Sep '13

25 Sep '13

To developers of pypy/lang-scheme, I wonder if anyone still maintains lang-scheme on bitbucket? I'm a university student facing a project for my compilers course, and would like to use lang-scheme as a starting point. Thanks.

3 2

merging pypy-pyarray branch
by Matti Picus 23 Sep '13

23 Sep '13

I am getting close to being ready to merge the pypy-pyarray branch. I related to all the review items from https://bitbucket.org/pypy/pypy/src/1a379a6f127bd6df8a4b628575c7124e83007e6… <https://bitbucket.org/pypy/pypy/src/1a379a6f127bd6df8a4b628575c7124e83007e6…> Does anyone have objections to merging pypy-pyarray into default (assuming the buildbots come up with no failures)? Executive summary - I took over Stefan__'s work to implement the c api interface from numpy. This together with a fork of numpy and a fork of matplotlib got him to a point where he could use non-interpretive plotting (png, pdf) in matplotlib. Matti

4 4

What a VM does
by Carl Friedrich Bolz 17 Sep '13

17 Sep '13

http://www.youtube.com/watch?v=G30vNCmMP1k Just remembered that that's the video that Dave Ungar (the main person behind the Self project) pointed me to when I met him for the first time, telling me that that's pretty much exactly what a VM does: Making it appear that something simple is happening while internally a lot of complex mechanisms are at work to uphold the illusion. Cheers, Carl Friedrich

1 0

pypy-stm - how do I know when transactions are turning inevitable?
by Костя Лопухин 15 Sep '13

15 Sep '13

Hello! As far as I understand, pypy-stm reports only conflicts, but not transactions that turn inevitable. But if transaction turns inevitable, it means that other transactions can not proceed. So it is hard to debug loss of parallelization when some transactions turn inevitable but other transactions do not conflict - the program is just silently slow. Is this correct? Are there any plans on giving some hooks for this case, or stacktraces similar to aborting transactions? Or maybe there is some other method?

2 1

Cannot use MacPorts PyPy to Translate pypy-c
by Jean-François Caron 11 Sep '13

11 Sep '13

Hi, I am following the instructions here: http://doc.pypy.org/en/latest/cppyy.html#installation and here: http://root.cern.ch/phpBB3/viewtopic.php?f=14&t=17018 In the first link, it says I can use a "built-in pypy" to do the translation to make it go faster. I have pypy 2.1.0 from MacPorts, but it fails with the following message: <<< jfcaron@jfcaron-MacBook:~/Projects/PyPyRoot/pypy$ pypy rpython/translator/goal/translate.py --opt=jit pypy/goal/targetpypystandalone.py --withmod-cppyy Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "rpython/translator/goal/translate.py", line 89, in <module> log = py.log.Producer("translation") File "/Users/jfcaron/Projects/PyPyRoot/pypy/py/_apipkg.py", line 114, in __makeattr result = importobj(modpath, attrname) File "/Users/jfcaron/Projects/PyPyRoot/pypy/py/_apipkg.py", line 37, in importobj module = __import__(modpath, None, None, ['__doc__']) File "/Users/jfcaron/Projects/PyPyRoot/pypy/py/_log/log.py", line 184, in <module> setattr(Syslog, _prio, getattr(py.std.syslog, _prio)) File "/Users/jfcaron/Projects/PyPyRoot/pypy/py/_std.py", line 13, in __getattr__ m = __import__(name) File "/opt/local/lib/pypy/lib_pypy/syslog.py", line 68, in <module> lib = ffi.verify(""" File "/opt/local/lib/pypy/lib_pypy/cffi/api.py", line 311, in verify lib = self.verifier.load_library() File "/opt/local/lib/pypy/lib_pypy/cffi/verifier.py", line 68, in load_library self.compile_module() File "/opt/local/lib/pypy/lib_pypy/cffi/verifier.py", line 55, in compile_module self._write_source() File "/opt/local/lib/pypy/lib_pypy/cffi/verifier.py", line 117, in _write_source file = open(self.sourcefilename, 'w') IOError: [Errno 2] No such file or directory: '/opt/local/lib/pypy/lib_pypy/__pycache__/_cffi__g7019d5d3xad93c709.c' >>> I am now doing the slow translation with CPython (also from MacPorts) with no problems so far. Is this just a version mismatch between the MacPorts pypy and that expected by the pypy on mercurial? Jean-François

4 6

LLVM next steps
by Armin Rigo 11 Sep '13

11 Sep '13

Hi Manuel, We've been suitably impressed by the results on the new llvm backend during the sprint (well, or suitably un-impressed by both gcc and clang's failure to reconstruct the SSA meaning of the C code). The current issue seems to be debugging. It would be nice if gdb presented at least source ".ll" code rather than just the assembler instructions --- actually, it would be more than nice: debugging at the level of assembler is a no-no. I was thinking: would it make sense to emit from the translation toolchain some files that are not in ".ll" format, but that are more or less a straightforward text file representation of the flow graphs and constants, and then have a separate tool that converts these files (seen as source) into .ll files? There are some advantages of doing it this way, even if it looks like Yet Another intermediate step. The first advantage is that it would give some inspectable files, which we could also request during testing for cases where the pygame flow graph inspector doesn't really work (e.g. too many graphs). Another advantage is that we might refactor the C backend to also be a separate tool that inputs these flowgraph text files, minimizing the amount of duplicate work between the C and the LLVM backends. And of course, the point is that the flowgraph-to-".ll" conversion would insert file line numbers, so that we can debug it in gdb seeing the flowgraph source lines. (I can even think about hacks to do the same even if we go via C...) Of course the drawback is that it's some non-trivial refactorization. Does it make sense? A bientôt, Armin.

5 7

ndarray cpyext api on the pypy-pyarray branch
by Matti Picus 10 Sep '13

10 Sep '13

I reverted the changes I made to the pypy-pyarray branch that changed c-api functions like PyArray_NDIM(arr). The original code had no real answer to what happens if these are called when arr is not an ndarray. While the reviewers of the branch suggested that these should raise an exception if arr is not a ndarray, Amaury Forgeot d'Arc convinced me that we should be consistent with numpy (perhaps dangerous) practice since in numpy these functions are c-defined macros with no type checking and the numerous example uses scattered through the numpy source code as well as scipy contain no error checking. In fact error checking is not possible since there is no clear idea of what an error condition in the return value would look like. The discussion we had on IRC starts here http://www.tismer.com/pypy/irc-logs/pypy/pypy.2013-09-08.log.html#t22:15 so if my explanation is unclear please read the log. Any objections to keeping with the original numpy-compatible dangerous behaviour? Matti

3 2