Mailman 3 October 2013 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus 22 Dec '13

22 Dec '13

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4

pip install rbtree fail in pypy env!
by KaShining 02 Nov '13

02 Nov '13

pypy 2.1 pip install rbtree >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Downloading/unpacking rbtree Running setup.py egg_info for package rbtree Installing collected packages: rbtree Running setup.py install for rbtree building 'rbtree' extension cc -O2 -fPIC -Wimplicit -I./src -I/pypy/include -c src/rbtree_impl.c -o build/temp.linux-x86_64-2.7/src/rbtree_impl.o cc -O2 -fPIC -Wimplicit -I./src -I/pypy/include -c src/rbtree.c -o build/temp.linux-x86_64-2.7/src/rbtree.o src/rbtree.c: In function '__pyx_f_6rbtree_6rbtree_byOffset': src/rbtree.c:2427:48: error: 'PyRange_Type' undeclared (first use in this function) src/rbtree.c:2427:48: note: each undeclared identifier is reported only once for each function it appears in src/rbtree.c: In function '__Pyx_GetException': src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4996:11: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4997:11: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4998:11: error: 'PyThreadState' has no member named 'exc_traceback' error: command 'cc' failed with exit status 1 Complete output from command /pypy/bin/pypy -c "import setuptools;__file__='/pypy/build/rbtree/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-s2EnAa-record/install-record.txt --install-headers /pypy/include/site/python2.7: running install running build running build_ext building 'rbtree' extension cc -O2 -fPIC -Wimplicit -I./src -I/pypy/include -c src/rbtree_impl.c -o build/temp.linux-x86_64-2.7/src/rbtree_impl.o cc -O2 -fPIC -Wimplicit -I./src -I/pypy/include -c src/rbtree.c -o build/temp.linux-x86_64-2.7/src/rbtree.o src/rbtree.c: In function '__pyx_f_6rbtree_6rbtree_byOffset': src/rbtree.c:2427:48: error: 'PyRange_Type' undeclared (first use in this function) src/rbtree.c:2427:48: note: each undeclared identifier is reported only once for each function it appears in src/rbtree.c: In function '__Pyx_GetException': src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4993:5: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4994:5: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4995:5: error: 'PyThreadState' has no member named 'exc_traceback' src/rbtree.c:4996:11: error: 'PyThreadState' has no member named 'exc_type' src/rbtree.c:4997:11: error: 'PyThreadState' has no member named 'exc_value' src/rbtree.c:4998:11: error: 'PyThreadState' has no member named 'exc_traceback' error: command 'cc' failed with exit status 1 ---------------------------------------- Command /pypy/bin/pypy -c "import setuptools;__file__='/pypy/build/rbtree/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-s2EnAa-record/install-record.txt --install-headers /pypy/include/site/python2.7 failed with error code 1 in /pypy/build/rbtree Storing complete log in /home/chenjiaming/.pip/pip.log

4 8

Question on installing pypy and the source code ..
by Dave Watson (davewats) 01 Nov '13

01 Nov '13

Hi Developers .. I have been looking at trying to install the pypy interpreter from source code as suggested from the: http://pypy.org website for the very first time. We don't have any previous version of the pypy interpreter installed on our systems at Cisco .. I ran into a bit of an issue in trying to follow the instructions. As was suggested I tried to use the "building from Source" code instructions as I wanted to build it from scratch as we have a somewhat unique environment, so it is usually better to try and build the tools from the source code .. All the build instructions seem to assume that you have a previous version of pypy already installed, in my case, I don't have a previous version of pypy already installed and am trying to create the first one.. :( .. I there a set of instructions for very first time builders ?? Thanks for any help that you can offer .. Cheers Dave W David Watson STEP Tools Master Cisco System<http://www.cisco.com/> 2000 Inovation Drive Ottawa, On K2W 3E8 PH: 613 254-3377 [cid:image001.jpg@01CDC1A7.C1732360]

2 4

RPython Environment Variables
by Ryan Gonzalez 31 Oct '13

31 Oct '13

I am writing a tool in RPython that needs to access environment variables. Now, all has been going well, but I noticed something interesting. The RPython translator sees os.environ as constant. But, will it still be constant on other platforms? What I'm saying is, are the environment variables permanently embedded into the application? i.e., if I run it on another PC, the values will be the same? Or is there an RLib equivalent? Because I really don't feel like writing a C wrapper... -- Ryan

3 2

Re: [pypy-dev] Question on installing pypy and the source code
by Mark Roberts 31 Oct '13

31 Oct '13

I've had really good luck installing pypy and development environments with pyenv. It includes recipes for installing pretty much any version of Python you might need. -Mark > Hi Developers .. > > I have been looking at trying to install the pypy interpreter from source code as suggested from the: http://pypy.org website for the very first time. We don't have any previous version of the pypy interpreter installed on our systems at Cisco .. > > I ran into a bit of an issue in trying to follow the instructions. As was suggested I tried to use the "building from Source" code instructions as I wanted to build it from scratch as we have a somewhat unique environment, so it is usually better to try and build the tools from the source code .. > > All the build instructions seem to assume that you have a previous version of pypy already installed, in my case, I don't have a previous version of pypy already installed and am trying to create the first one.. :( .. > > I there a set of instructions for very first time builders ?? > > Thanks for any help that you can offer .. > > Cheers > > Dave W > > > David Watson > STEP Tools Master > Cisco System<http://www.cisco.com/> > 2000 Inovation Drive > Ottawa, On K2W 3E8 > PH: 613 254-3377 > [cid:image001.jpg@01CDC1A7.C1732360]

2 1

numpy repo under pypy on bitbucket
by Matti Picus 30 Oct '13

30 Oct '13

Thanks to Brian Kearns, the pypy-compat branch of the numpy repo at https://bitbucket.org/pypy/numpy <https://bitbucket.org/pypy/numpy/> is starting to become the prefrered method of using numpy on pypy. Note this is a git repo. All commiters to pypy should have a commit bit here, let me know if something needs to be changed. Also, I enabled a commit hook so commits should appear in IRC, if this gets excessive it can be turned off. Waiting for your contributions, Matti <https://bitbucket.org/pypy/numpy/>

2 1

trying out STM for some numbers on more cores
by wlavrijsen＠lbl.gov 30 Oct '13

30 Oct '13

Hi, last week I inherited a dual 8 core E5-2650 box, so 16 physical cores. For fun I wanted to redo the STM numbers from the blog posting on more cores. I killed the printout coming from rpython, as besides giving unnecessary syncing, it also causes crashes beyond 12 threads or so. Here are my numbers: Reference CPython, 1 thread: 271.17 Reference PyPy-2.1, 1 thread: 9.63 (Note above for whatever reason my CPython run is slower that in the blog post, even as the pypy run is a lot faster?) Freshly-translated (1385fb758727+ (stmgc-c4)): richards.py, 100 iterations each, printout from rpython removed: Threads avg. time speedup 1 358.48 1.0 2 244.86 1.5 4 162.87 2.2 8 141.48 2.5 16 127.43 2.8 32 146.57 2.4 Note that the speedup numbers above are less than what was posted on the blog, and scaling with hyperthreads isn't working well. Has any attempt been made to pin threads? Going beyond 32 crashes in: #1 0x0000000000d29026 in pypy_g_start_new_thread () #2 0x0000000000498347 in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_W_Root_W_R () #3 0x000000000085a8c1 in pypy_g_BuiltinCode_funcrun_obj () #4 0x0000000000858255 in pypy_g_funccall_valuestack__AccessDirect_None () #5 0x0000000000d73d86 in pypy_g_CALL_METHOD__AccessDirect_star_1 () #6 0x000000000089d3a3 in pypy_g_dispatch_bytecode__AccessDirect_None () #7 0x00000000008a34b5 in pypy_g_handle_bytecode__AccessDirect_None () #8 0x0000000000cfe311 in pypy_g_dispatch__AccessDirect_None_stm () #9 0x000000000136f974 in pypy_g__stm_callback_4 () #10 0x00000000014ea1cf in stm_perform_transaction () #11 0x000000000050e62e in pypy_g_ll_invoke_stm__pypy_objspace_std_frame_StdObjSpa () #12 0x00000000011ec935 in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () #13 0x0000000000874bda in pypy_g_PyFrame_run () Still trying to see whether I can get PyPy to run on the MIC. :) Best regards, Wim -- WLavrijsen(a)lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

5 11

A few notes trying to compile pypy
by Kevin Burke 29 Oct '13

29 Oct '13

Hey everyone, I was trying to compile pypy and made the following mistakes: - On this page: http://pypy.org/download.html I couldn't figure out whether I was supposed to run all three of these commands: pypy ../../rpython/bin/rpython -Ojit targetpypystandalone # get the JIT version pypy ../../rpython/bin/rpython -O2 targetpypystandalone # get the no-jit version pypy ../../rpython/bin/rpython -O2 --sandbox targetpypystandalone # get the sandbox version or just one. - Once I got it installled I didn't realize there was an additional step after pypy-c was created. - I tried running python package.py -h and that didn't do what I expected (show help dialog) - I tried running package.py with the right options and it hung with this message (on both Mac and Centos): sll_os.ll_os_getenv(sPYPY_GENERATIONGC_NURSERY The only Google search results for this are an IRC channel from 2 years ago. It turned out I didn't want the Sandbox option, I think. - I tried compiling outside the sandbox and got this error: raise NoTTY("Cannot start the debugger when stdout is captured.") It turns out this is a red herring and the real error was way above that: error: ffi.h: No such file or directory It would be nice if these were more obvious.. of course I expect to run into issues but the errors could be more clear when they are. Continuing the install process now, will post updates here as I have them. I am happy to help try and fix these where I can, if you can point me in the right direction - I have a fair amount of experience working with new users at Twilio. I have also made exactly one commit to Pypy source code: https://bitbucket.org/pypy/pypy/commits/665baf750859/ Pypy is a lovely project and I'd love to try and help make this process easier. Best, Kevin -- Kevin Burke | Twilio phone: 925.271.7005 | kev.inburke.com

7 12