Mailman 3 December 2011 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

PyPy/NumPyPy sprint(s) in Leipzig, Germany?
by Mike Müller 20 Feb '12

20 Feb '12

Hi, I am just wondering if anybody is interested in sprinting on PyPy and in particular NumPyPy in Leipzig sometime in 2012. I can offer working space for up to 12 people with Wi-Fi as well as some basic catering (hot and cold drinks, snacks, pizza etc.) for a few days (up to a week). Accommodation in Leipzig is very reasonably priced. For example, there is a decent hotel very close to the venue for 35 Euros/night (single) or 50 Euros/night (double) including breakfast. Being a logistics center, Leipzig is easy to travel to by car, train or airplane including budget airlines. I would also act as co-sponsor (with the resources stated above) for an application for sprint funds (http://pythonsprints.com/cfa/ or other sources) that could cover (parts) of the traveling and accommodation expenses. Let me know what you think about it. I am open to ideas. Mike

5 31

How to turn a crawling caterpillar of a VM into a graceful butterfly
by Laurence Tratt 01 Jan '12

01 Jan '12

Hi all, As many of you know, over the past few months I've been creating an RPython VM for the Converge language <http://convergepl.org>. This is now mostly complete at a basic level - enough to run pretty much all of my Converge programs at least on Linux and OpenBSD. [Major remaining issues are: no floating point numbers; 32 bit support is not finished; may not compile on OS X.] First, the good news. The RPython VM (~3 months effort) is currently close to 3 times faster than the old C-based VM (~18 months effort). I think the Converge VM is the first medium-scale VM to be created by someone outside the core PyPy group, so those numbers are a testament to the power of the RPython approach. I'd like to thank (alphabetically) Carl Friedrich Bolz, Maciej Fijalkowski, and Armin Rigo who've offered help, encouragement and (in Armin's case) big changes to RPython to help the Converge VM. I wouldn't have got this far without their help, or of others on the PyPy IRC channel. Now, the bad news: the RPython VM should be a lot faster than it currently is as the old VM is (to say the least) not very good. In large part this is because I simply don't know how best to optimise an RPython VM, particularly the JIT (in fact, at the moment the JITted VM seems to be sometimes slower than the non-JIT VM: I don't have an explanation for why). I'm therefore soliciting advice / code from those more knowledgeable than myself on how to optimise an RPython VM. Note that I'm intentionally being more general than the Converge VM: I hope that some of the ideas generated will prove useful to the many VMs that I hope will come to be implemented in RPython. Some of my ignorance is simply that many parts of RPython have little or no documentation: by playing the part of a clueless outsider (it comes naturally to me!), I hope I may help pinpoint areas where further documentation is most needed. If you want to test out the VM it's here: https://github.com/ltratt/converge Building should mostly be: $ export PYPY_SRC=<location of PyPy source directory> $ cd converge $ ./configure $ make That will build a JITted VM. If you want a lower level of optimisation, specify it to configure e.g. "./configure --opt=3". Please note that, for the time being, on Linux the JIT won't work with the default GC root finder, so you'll need to manually specify --gcrootfinder=shadowstack in vm/Makefile. Apart from that, building on 64 bit Unix should hopefully be reasonably simple. A simple performance benchmark is along the lines of "make clean ; cd vm ; make ; cd .. ; time make regress" which builds the compiler, various examples, and the handful of tests that Converge comes with. [Please note, Converge wasn't built with a TDD philosophy; while I welcome contributions of tests, I am unable to make any major efforts in that regard myself in the short-term. I know this sits uneasily with many in the PyPy community, but I hope you are able to overlook this difference in development philosophy.] Here are some examples of questions I'd love to know answers to: * Why is the JITted VM that's built sometimes 2x slower than --opt=3, but other times a few percent faster *on the same benchmark*?! * What are virtualrefs? * What is a more precise semantics of elidable? * What is 'specialize'? * Is it worth manually inlining functions in the main VM loop? * Can I avoid the (many) calls to rffi.charp[size]2str, given that they're mostly taken from the never-free'd mod_bc? Would this work well on some GCs but not others? No doubt there are many other things I would do well to know, but plainly do not - please educate me! If you have any questions or comments about Converge or the VM, please don't hesitate to ask me - and thank you in advance for your help. Yours, Laurie -- Personal http://tratt.net/laurie/ The Converge programming language http://convergepl.org/ https://github.com/ltratt http://twitter.com/laurencetratt

4 11

micronumpy var & std patch
by mike c 31 Dec '11

31 Dec '11

Hi All, Find attached a patch (to default branch) to add var and std to micronumpy. Two simple tests were added, and all the micronumpy module tests still pass. Feedback is welcome. Not sure if there's a better way of nesting the method calls in descr_var() - currently I have to do the calculation a bit at a time and after each calculation assert that what I get back is actually a BaseArray. Not sure if this should go to the mailing list or to bugs.pypy.org. Hope here is ok. Mike.

2 1

Leysin Winter Sprint
by Armin Rigo 30 Dec '11

30 Dec '11

===================================================================== PyPy Leysin Winter Sprint (15-22nd January 2012) ===================================================================== The next PyPy sprint will be in Leysin, Switzerland, for the eighth time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. ------------------------------ Goals and topics of the sprint ------------------------------ * Py3k: work towards supporting Python 3 in PyPy * NumPyPy: work towards supporting the numpy module in PyPy * JIT backends: integrate tests for ARM; look at the PowerPC 64; maybe try again to write an LLVM- or GCC-based one * STM and STM-related topics; or the Concurrent Mark-n-Sweep GC * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski. ----------- Exact times ----------- The work days should be 15-21 January 2011 (Sunday-Saturday). The official plans are for people to arrive on the 14th or the 15th, and to leave on the 22nd. ----------------------- Location & Accomodation ----------------------- Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connexion with wireless installed. You can of course arrange your own lodging anywhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-) Please *confirm* that you are coming so that we can adjust the reservations as appropriate. The rate so far has been around 60 CHF a night all included in 2-person rooms, with breakfast. There are larger rooms too (less expensive) and maybe the possibility to get a single room if you really want to. Please register by Mercurial:: https://bitbucket.org/pypy/extradoc/ https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2… or on the pypy-dev mailing list if you do not yet have check-in rights: http://mail.python.org/mailman/listinfo/pypy-dev You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around -- bring a EU-format power strip if you have one.

2 2

Use of marshal in the sandbox: is stdlib marshal OK?
by Ned Batchelder 29 Dec '11

29 Dec '11

The sandbox uses pypy's own implementation of marshal. In pypy/translator/sandbox/sandlib.py is this comment: # Note: we use lib_pypy/marshal.py instead of the built-in marshal # for two reasons. The built-in module could be made to segfault # or be attackable in other ways by sending malicious input to # load(). Also, marshal.load(f) blocks with the GIL held when # f is a pipe with no data immediately avaialble, preventing the # _waiting_thread to run. I'd like to remove as many dependencies as possible from the sandbox code, so I'd like to explore the possibility of using the standard library marshal module. The first reason above is about crashing marshal with malicious input. To my thinking, we are in control of what data is marshaled, so we don't have to worry about malicious input. The untrusted Python code running in the sandbox doesn't have a way of sending marshaled data, so we don't have to worry that it will be used to attack the marshal module. The stdout of the untrusted Python code will become a string that is marshaled, but that doesn't provide a way for the untrusted code to attack the marshal module. Or have I missed something? The second reason I can't address, is this still a problem? What bad effects will we see if it is? --Ned.

4 6

My experience with PyPy
by Michael Sioutis 24 Dec '11

24 Dec '11

Hello! I haven't been following the list, cause I joined yesterday in order to share my impressions of PyPy here, so I don't know if these kinds of posts are frequent and annoying. I'm using PyPy the last few weeks in a qualitative spatial reasoner I am developing. The speedup is up to 10 times over CPython, and it grows even more as the input size grows bigger, but that's not what impressed me the most, because I was expecting that. You can find in my presentation here<http://cgi.di.uoa.gr/~sioutis/files/PyRCC8.pdf> at slides 25 and 33 some comparison diagrams against C (red color) and C++ (blue color) implementations of analogous reasoners. I am using different data structures and slightly modified algorithms, so you should not consider that the comparison diagrams is upon the exact same piece of code. Long story short, you will find out that the pypy reasoner ranks in positions 1 and 2 respectively, and I believe it will also rank 1 in the second diagram if I grow the input even more (I currently don't have enough memory and the C implementation tops it at 900 nodes, so it would be unreasonable and unfair to run on my own). The impressive part of both diagrams is the scalability pypy offers me. It starts slower than the statically compiled languages, but the more you push the input sizes, the faster it goes compared to the other two. I started developing the reasoner in python not necessarily to be fast in timings, but because I want to go beyond the state of the art in qualitative spatial reasoning and present sth new, so python was the language of choice to do it fast, in terms of working hours :) This is future work, that will be based on the current reasoner. *Thank you* for being patient, if you actually took the time to read my story, and have a Merry Xmas! Mike

2 1

Generators in RPython
by Armin Rigo 23 Dec '11

23 Dec '11

Hi all, RPython now has support for generators. The support is minimal for now: * only pre-Python-2.5 generators: no .send(), no "yield" inside a "try:finally:" block, etc. * you cannot use a "for" loop to iterate over a generator-iterator. You have to use explicitly .next() and catch the StopIteration, for now. * two different generators produce generator-iterator objects that the annotator cannot unify, even if they happen to yield objects of the same type. You have to work around that limitation e.g. as shown in rpython/test/test_generator, test_cannot_merge. I still hope that this is enough to be useful to the Prolog and Converge interpreters :-) A bientôt, Armin.

2 1

Pypy memory usage
by Colin Kern 22 Dec '11

22 Dec '11

Hi all, I have a program that uses the multiprocessing package to run a set of jobs. When I run it using the standard python interpreter on my computer, top shows 8 threads each using about 40M of memory, which remains fairly steady. When I use pypy, both 1.6 and 1.7, the 8 threads almost immediately show using 90-100M of memory, and then that continues to climb as the program runs. Each job runs a lot faster in pypy, but usually before all the jobs are done, the memory on the system is exhausted and swapping starts, which brings the execution speed to a crawl. Is this something anyone else has experienced? Thanks, Colin

2 1

Sprint in Leysin?
by Armin Rigo 22 Dec '11

22 Dec '11

Hi all, Who would be interested in the next sprint being in Leysin? Some time around the 2nd half of January? A bientôt, Armin.

4 5