Mailman 3 August 2011 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor March 10, 2014

March 10, 2014

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

Building pypy on freebsd 64bit
by Shadrin Eugene Nov. 18, 2011

Nov. 18, 2011

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

Updated 'High Performance Python' tutorial (the one from EuroPython 2011)
by Ian Ozsvald Nov. 16, 2011

Nov. 16, 2011

Dear all, I've published v0.2 of my High Performance Python tutorial write-up from the session I ran at EuroPython: http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from… Antonio - you asked earlier if the 'expanded math' version of the Mandelbrot solver (using doubles rather than complex numbers) would be faster - I've timed it and it is a bit faster with a nightly build of PyPy, but nowhere near as fast at ShedSkin's generated C output (details below). Maciej - thanks for pointing me at the numpy module. I've added a tiny section showing numpy in PyPy but I haven't converted the Mandelbrot solver to use it (even finishing v0.2 took longer than I'd thought). I'm hoping that some more exposure in the report might bring in more volunteers from outside. Here's a clip from the report in the PyPy section: "By running pypy pure_python.py 1000 1000 on my MacBook it takes 5.9 seconds, running pypy pure_python_2.py 1000 1000 takes 4.9 seconds. (Ian - the only difference with pure_python_2.py is that local dereferences in the tight loop are moved outside the loop, causing fewer dereference operations) As an additional test (not shown in the graphs) I ran pypy shedskin2.py 1000 1000 which runs the expanded math version of the shedskin variant below (this replaces complex numbers with floats and expands abs to avoid the square root). The shedskin2.py result takes 3.2 seconds (which is still much slower than the 0.4s version compiled using shedskin)." The pure_python src is here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/… shedskin2.py is available here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/… I haven't tested whether the warm-up periods for PyPy are significant, possibly they account for much of the difference between ShedSkin and PyPy? I want to revisit this but for the next few weeks I have to go back to other projects. I hope the report brings in some new folk for PyPy, Ian. -- Ian Ozsvald (A.I. researcher, screencaster) ian(a)IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald

5 8

Anyone interested in a MIPS port...
by Vishal Oct. 16, 2011

Oct. 16, 2011

Hello, I would like to collect volunteers for a MIPS port of PyPy JIT. MIPS is used across the embedded scene universally and a PyPy support for MIPS would be wonderful. So I have 2 questions (may seem like very nuby like...pardon me for that): a) Does it make sense to have a MIPS port of the PyPy JIT. b) How much hardware dependent is a JIT port? Thanks and best regards, Vishal Sapre

5 5

Stacklets
by Armin Rigo Sept. 28, 2011

Sept. 28, 2011

Hi all, Some progress report from my side: I am working on "stacklets", which is, as the name implies, similar to Stackless. It is a rewrite of tealets, which was itself a greenlet-like replacement for Stackless Python. (Yes, "tealet" because of "greenlet": green tea... last time I can do the joke, though, as tealets go away.) Basically a "stacklet" is just a one-shot continuation. The name "stacklet" is going to be internal; at app-level it's just called a continuation. It's the most basic primitive I could eventually arrive at, which is not surprising at all, given that it *is* well known to be the most basic primitive in functional programming. See for example http://en.wikipedia.org/wiki/Call-with-current-continuation. (Of course it is possible to emulate it using other primitives, e.g. http://sigusr2.net/2011/Aug/09/call-cc-for-python.html, but that's abstraction inversion, really.) In the stacklet branch you get a new built-in module called "_continuation" which has only one function: "new", also exposed under the name "callcc", for the reference to various functional programming languages. Doing "_continuation.new(func, args...)" calls "func(c, args...)" with 'c' being a new Continuation object. Calling "c.switch()" will switch back to the point where 'c' was created, and this point will return another continuation 'c2' that can be used to come back again. After a number of switches the function func() can return; it should return yet another Continuation, the one to jump to next. - Continuations are objects with a "switch()" method instead of directly callables, because that seemed more Pythonic - One shot: once you resume a continuation, the object cannot be switched to any more - Garbage collectable: if a pending continuation dies, then every resource reachable only by resuming it dies too - Support for exceptions: if func() terminates by raising an exception, the exception is propagated in the parent (see below) - Implemented like greenlets by moving around part of the C stack, which is easily compatible with the JIT - See http://en.wikipedia.org/wiki/Call-with-current-continuation for a list of what can easily be built on top of them. Once I'm done refactoring the GC to support them (in particular with shadowstack, which requires a bit of care), I will write a greenlet.py module at app-level implementing full greenlets (which should be less than 100 lines of code). It should be useful as a demo of how higher-level abstractions can be implemented on top of continuations. Ah, I should also write a CPython wrapper around the core, which is already written in C, to expose the same continuations for CPython. One (longish, sorry) word about propagating exceptions: this is not usually a concern of functional programming languages, so I had to come up with a way to define where an exception goes. It goes to the 'parent', which is defined in a greenlet-like way: the parent of a 'func' is the piece of C stack that originally started 'func'. I think that this way is the most correct one. Let me explain. One goal of stacklets is to give a "composable" concept, where "composable" is defined as: if two pieces of code independently use continuations, and if both pieces work fine when run in isolation, then they should work fine when run together too. Stackless Python's tasklets are composable: you don't have actually references to tasklets, but only to channel objects, which does the trick (I can give more details if anyone is interested). Greenlets on the other hand are not, and neither are coroutines. Basic continuations (without exceptions) are composable again. Internally, even with pypy+stacklets, continuations form a tree similar to greenlets, but this tree is not exposed to app-level, because doing so would break composability. For example, you could ask the question "what is my 'parent' piece of stack" and jump there; to see how this can be wrong, consider the following call chain: e() -> _continuation.new() -> f() -> g() -> h() where g() is in some other module but calls back your own h(). In this case the 'parent' piece of stack is not well-defined: h() would expect that it is e(), but maybe g() internally did another _continuation.new() too. However, in case of exceptions, then it is *still* the correct thing to do to propagate the exception into the parent. If h() raises, it is propagated back to g(), back to f(), and back to e(), even if g() internally did another _continuation.new(). Every level is free to capture the exception or let it propagate if it's of an unknown class, just like normally. So h() might raise an exception and expect e() to catch it, which will work just like it does in the normal case: it will work as long as g(), which is in another module, doesn't explicitly catch exceptions of the same class as well. I suppose this is already well-known, but I couldn't find an obvious reference. (For example, looking at Ruby doesn't help much because its callcc seems slightly different.) A bientôt, Armin.

6 16

Re: [pypy-dev] bounties for pypy
by Laura Creighton Sept. 16, 2011

Sept. 16, 2011

>The idea was also to possibly attract new developers ... for example, if >there would be "10 days in money" for adapting py2exe, I am sure many wou >ld >jump to solve this puzzle. This is sort of a bad example. Because py2exe embeds CPython, and we wouldn't want to do that. So what we would probably want to do is to make some general tool that willmake a windows binary, or a mac one, and get rid of the need for bzfreeze and friends. So now you are looking at a general embedding solution, and that is more than 10 days worth of work. But I get the idea. <snip> >> my dream was of a trustee service: after somebody commits to do the wor >k, >the pledgers have to pay to a trustee. then the work is done. then the >trustee pays the worker. This is one of the things I want to talk with fundedbyme about. But having an explicit trustee is a new idea. I think the pypy core developers are already rather well trusted in this community, but this may be important to new developers who aren't as well known. And it handles the problem' of 'I got sick and cannot do this any more' more gracefully than other solutions. >Hmmm.... a structure could be: > >- service provider does the technical stuff, as in: > # website > # collect pledges > # handle project description > # collect money > # distribute money after feature completion fundedbyme has sort of indicated an interst in doing this (except they were talking about distribution before, and I was leaving project description to the project, not outsiders). I will follow up on this when I get back home to Sweden. >- PSF / pypy-foundation / whateverfoundation provides the trust > >Thanks for confirming the need for such a thing! > >Harald Thanks once again for seeing a marketing solution that nerds like us often miss. Laura

6 12

Re: [pypy-dev] Moving the project forward
by Jesse Noller Sept. 8, 2011

Sept. 8, 2011

I've put up a splash page for the project this AM: http://speed.python.org/ jesse

6 10

Status of ARM backend
by Eric van Riet Paap Sept. 5, 2011

Sept. 5, 2011

Hi, Can someone please give us an update on the status of the ARM backend. Is it time already to get our iPhones and Androids out of our pockets and start creating that killer app using PyPy? cheers, Eric

5 4

your thoughts on low level optimizations
by Gertjan van Zwieten Sept. 3, 2011

Sept. 3, 2011

Dear all, Following today's blog post [1] about wrapping C++ libraries, I would like to take the opportunity to get some opinions from the PyPy community on a related topic. I hope this mailing list is the appropriate place for this sort of discussion. My name is Gertjan van Zwieten and I have been a Python enthusiast since long. I have used it extensively in the context of my PhD project, which was related to Finite Element modelling, and I have recently taken steps together with a friend of mine towards continuing on this path commercially [2]. We intend to design our own Finite Element framework, which we aim to be open source, written largely in Python. It is the 'largely' that I would like to discuss here. I have always argued to people that, yes, it is possible to do computationally intensive things in an interpreted language, as long as the majority of work is done by compiled components. For me the obvious example of that is Numpy, which provides a rather specific data type and a set of vectorized operations, as well as an interface with optimized linear algebra libraries. This leaves Python as a convenient and very powerful glue language to connect these optimized components into something purposeful. With that conceptualization in mind, my preferred route for optimizing code was to identify critical components of generic nature, and re-implement them as a plain C module using the Python API. So this is how I used to look at things. I would say that things were nice and clear this way, until PyPy started throwing stones in the water. I have not so far used PyPy in any actual application, mainly (if not only) because of lacking Numpy support. But I can see it holds great promise, especially for the computational community (I am also eagerly following the transactional memory developments) and I anticipate making the switch after it matures a little bit further. However, that does put me in the situation that, currently, I am not sure anymore what things to implement in C, if any, and in what form. Right now I am prototyping our framework using only Numpy and pure Python. I avoid premature optimization as much as possible, and in fact at times find myself sacrificing speed for elegance, arguing that I will be able to bring back efficiency later on in a deeper layer. For example, rather than adopting a numbering scheme for the many computational elements such as is common in Finite Element implementations, I have moved to a more object oriented approach that is more flexible and allows for more error checking, but that forces me to put in dictionaries what used to be Numpy arrays. Obviously dictionaries were not meant to be numeric data structures, and it's a typical component that I would try to implement efficiently in C eventually. With PyPy at the horizon, I am not so sure anymore. For one I'm not sure if PyPy will ever be able to use the C module, or use it efficiently; I could understand if support exists merely for sake of compatibility. I am also not certain if I should maybe forget about C altogether and rely on the JIT to compile the Python for loops that I have always tried to avoid. Would you go as far as to say that there will be no more reason for low level programming whatsoever? Or would you advise to write the component in RPython and use the translator to compile it? With my poor overview of these things, there are very few arguments that I can formulate in favour or against any of these options. Regarding today's blog post, I have the feeling that this is meant more for wrapping existing C++ libraries than for starting new ones, is that correct? Or if not, and it is, in fact, an option to consider, will this be able to work in CPython, too? That would make the transition a bit easier, obviously. I am very interested what your views are on this topic of optimizations. Best regards and thanks a lot for working on PyPy! Gertjan [1] http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.h… [2] http://hvzengineering.nl

5 9

Failing App-Level-Test test_posix2.py
by Juergen Boemmels Sept. 1, 2011

Sept. 1, 2011

Hi, Since some weeks in one Applevel-test is failing consitently on the buildbot: module/posix/test/test_posix2.py with the failure: if sys.platform.startswith('linux'): > assert hasattr(st, 'st_rdev') E assert hasattr(posix.stat_result(st_mode=33152,st_ino=1588275L, st_dev=64256L, st_nlink=1,s...integer_atime=1314759734, _integer_mtime=1314759734,_integer_ctime=1314759734), 'st_rdev') module/posix/test/test_posix2.py:135: AssertionError ================ 1 failed, 77 passed, 6 skipped in 4.80 seconds ================ So it seems that st_rdev is not available I tracked this problem down to a chicken-egg problem. The relevant portion is here in pypy/rpython/module/ll_os_stat.py: # for now, check the host Python to know which st_xxx fields exist STAT_FIELDS = [(_name, _TYPE) for (_name, _TYPE) in ALL_STAT_FIELDS if hasattr(os.stat_result, _name)] As pypy is meanwhile build with pypy, relying on the host python is not a good idea. A pypy without st_rdev will again build only a new pypy without st_rdev, even if the platform supports st_rdev. As an experiment I bootstrapped pypy again with python an the error disappered # python translate.py -O2 targetpypystandalone.py # ./pypy-c ../../../pytest.py ../../module/posix/test/test_posix2.py -A ========================= test session starts ========================== platform linux2 -- Python 2.7.1[pypy-1.6.0] -- pytest-2.1.0.dev4 pytest-2.1.0.dev4 from /home/boemmels/src/pypy/pytest.pyc collected 80 items ../../module/posix/test/test_posix2.py ...........................................................s............ ........ ================= 79 passed, 1 skipped in 5.07 seconds ================= This bootstrapped pypy-c is also cappable of rebuilding an pypy with st_rdev enabled. But this is not a clean solution. I think the way to go is to no longer rely on the Host-Python, but use the appropiate configure magic. Something like #ifdef HAVE_STRUCT_STAT_ST_RDEV but in a pythonic way. Unfortunatly I'm not familar with pypy's configuration system so I got stuck here. Can anybody tell me how to test for available struct members like AC_CHECK_MEMBERS([struct stat.st_rdev]) in autoconf? Regards juergen ___________________________________________________________ Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die Toolbar eingebaut! http://produkte.web.de/go/toolbar

2 1