Mailman 3 December 2013 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

arm-v7 buildbot down for the next two weeks
by Matti Picus 30 Dec '13

30 Dec '13

1 0

Leysin
by Johan Råde 29 Dec '13

29 Dec '13

Hi, I'm a software developer from Sweden. I would like to attend the Leysin Winter sprint. How do I register and reserve accommodation? Best regards, Johan Råde

4 5

[refactor-str-types] Translating the PyPy Python interpreter failed.
by Prasad Joshi 29 Dec '13

29 Dec '13

prasad@pjoshi:~/python/pypy/pypy/goal$ python ../../rpython/bin/rpython --opt=jit targetpypystandalone.py ... ... ... ... [Timer] Timings: [Timer] annotate --- 138.9 s [Timer] ========================================== [Timer] Total: --- 138.9 s [translation:info] Error: [translation:info] File "/home/prasad/python/pypy/rpython/translator/goal/translate.py", line 318, in main [translation:info] drv.proceed(goals) [translation:info] File "/home/prasad/python/pypy/rpython/translator/driver.py", line 534, in proceed [translation:info] return self._execute(goals, task_skip = self._maybe_skip()) [translation:info] File "/home/prasad/python/pypy/rpython/translator/tool/taskengine.py", line 114, in _execute [translation:info] res = self._do(goal, taskcallable, *args, **kwds) [translation:info] File "/home/prasad/python/pypy/rpython/translator/driver.py", line 283, in _do [translation:info] res = func() [translation:info] File "/home/prasad/python/pypy/rpython/translator/driver.py", line 320, in task_annotate [translation:info] s = annotator.build_types(self.entry_point, self.inputtypes) [translation:info] File "/home/prasad/python/pypy/rpython/annotator/annrpython.py", line 91, in build_types [translation:info] return self.build_graph_types(flowgraph, inputcells, complete_now=complete_now) [translation:info] File "/home/prasad/python/pypy/rpython/annotator/annrpython.py", line 145, in build_graph_types [translation:info] self.complete() [translation:info] File "/home/prasad/python/pypy/rpython/annotator/annrpython.py", line 199, in complete [translation:info] self.complete_pending_blocks() [translation:info] File "/home/prasad/python/pypy/rpython/annotator/annrpython.py", line 194, in complete_pending_blocks [translation:info] self.processblock(graph, block) [translation:info] File "/home/prasad/python/pypy/rpython/annotator/annrpython.py", line 344, in processblock [translation:info] self.flowin(graph, block) [translation:info] File "/home/prasad/python/pypy/rpython/annotator/annrpython.py", line 408, in flowin [translation:info] self.consider_op(block, i) [translation:info] File "/home/prasad/python/pypy/rpython/annotator/annrpython.py", line 602, in consider_op [translation:info] resultcell = consider_meth(*argcells) [translation:info] File "<3933-codegen /home/prasad/python/pypy/rpython/annotator/annrpython.py:641>", line 3, in consider_op_getattr [translation:info] return arg.getattr(*args) [translation:info] File "/home/prasad/python/pypy/rpython/annotator/unaryop.py", line 144, in getattr [translation:info] raise AnnotatorError("Cannot find attribute %r on %r" % (attr, obj)) [translation:ERROR] AnnotatorError: [translation:ERROR] [translation:ERROR] Cannot find attribute 'isspace' on SomeUnicodeCodePoint() [translation:ERROR] [translation:ERROR] [translation:ERROR] v324 = getattr(v323, ('isspace')) [translation:ERROR] [translation:ERROR] In <FunctionGraph of (rpython.rlib.rstring:89)rsplit__unicode at 0x1e5d0bd0>: [translation:ERROR] Happened at file /home/prasad/python/pypy/rpython/rlib/rstring.py line 98 [translation:ERROR] [translation:ERROR] ==> if not value[i].isspace(): [translation:ERROR] [translation:ERROR] Known variable annotations: [translation:ERROR] v323 = SomeUnicodeCodePoint() [translation:ERROR] [translation:ERROR] Processing block: [translation:ERROR] block@64 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> [translation:ERROR] in (rpython.rlib.rstring:89)rsplit__unicode [translation:ERROR] containing the following operations: [translation:ERROR] v323 = getitem(value_0, i_0) [translation:ERROR] v324 = getattr(v323, ('isspace')) [translation:ERROR] v325 = simple_call(v324) [translation:ERROR] v326 = bool(v325) [translation:ERROR] --end-- [translation] start debugger...

2 1

Kindly guide
by Jashan Goyal 26 Dec '13

26 Dec '13

hi, I am jashan goyal.I am interested in contributing to pypy please guide me how to begin. Thanks

2 1

Interested in GSoC 2014
by Prasad Joshi 25 Dec '13

25 Dec '13

Hello All, I am interested in participating GSoC 2014. I have went through last year's eligibility criterion, I think I am qualified to participate. I know GSoC 2014 still has more than 2-3 months, however I would like to start early. Please let me know suggestions. After going through GSoC 2013 page (https://wiki.python.org/moin/SummerOfCode/2013), I think I would be interested in Core Python or PyPy. Please let me know how should I start. Thanks a lot for your inputs. Thanks and Regards, Prasad

3 2

Re: [pypy-dev] why logging in pypy not need acquire lock?
by KaShining 24 Dec '13

24 Dec '13

ok ------------------ 原始邮件 ------------------ 发件人: "Armin Rigo" <arigo(a)tunes.org>; 发送时间: 2013年12月24日(星期二) 18:08 收件人: "KaShining" <mail2shine(a)qq.com>; 抄送: "pypy-dev" <pypy-dev(a)python.org>; 主题: Re: [pypy-dev] why logging in pypy not need acquire lock? Hi KaShining, On Tue, Dec 24, 2013 at 9:15 AM, KaShining <mail2shine(a)qq.com> wrote: > why logging in pypy not need acquire lock? > left from pypy whle right from cpython PyPy's stdlib is from CPython 2.7.3, where it doesn't acquire a lock either. It seems it was some bug that was discovered and fixed in a more recent 2.7.x. We'll get it too when we upgrade the CPython stdlib version in PyPy. A bient��t, Armin.

1 0

why logging in pypy not need acquire lock?
by KaShining 24 Dec '13

24 Dec '13

why logging in pypy not need acquire lock? left from pypy whle right from cpython

2 3

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus 22 Dec '13

22 Dec '13

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4