Mailman 3 January 2014 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

performance problems with Krakatau
by Robert Grosse 13 Apr '14

13 Apr '14

Hi, I have been trying to use Pypy to speed up the Krakatau decompiler ( https://github.com/Storyyeller/Krakatau). It is a large, pure Python application with several compute intensive parts, so I thought it would work well. Unfortunately, there is no clear speedup, and Pypy requires several times as much memory as well, making it unusual for larger inputs. For example, decompiling a quarter of ASM, I got the following results (execution time, memory usage) cpython 64 - 62.5s, 102.6kb cpython 32 - 69.2s, 54.5kb pypy 2.1.0 - 106.5s, 277.8kb pypy 2.2.1 - 109.2s, 194.6kb Sometimes, 2.2.1 is faster than 2.1.0, but they're both clearly much worse than CPython. These tests were performed on Windows 7 64bit using the prebuilt 32bit binaries of Pypy. I tested the 32bit version of CPython too, to see if the problem was a lack of 64bit support. However, CPython 32bit also vastly outperformed Pypy. Execution time was measured using time.time(). Memory usage was measured by watching the Windows Resource Manager and recording the peak Private value. Similar patterns were seen in Working Set, etc. I thought the increased memory usage at least might be explained by constant overhead from compiled code or from it not running long enough to trigger full garbage collection. However, Pypy continues to use several times as much memory on much larger examples. Does anyone know what could be going on here? Pypy isn't normally slower than CPython. Is there a way for me to tell what the problem is?

2 11

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

Translating PyPy on Windows
by Johan Råde 11 Feb '14

11 Feb '14

Hi everyone! When I translated PyPy on Windows, on a laptop with an Intel 3210M processor, I got the following timings: annotate --- 464.5 s rtype_lltype --- 882.7 s pyjitpl_lltype --- 825.9 s backendopt_lltype --- 259.2 s stackcheckinsertion_lltype --- 309.0 s database_c --- 349.6 s source_c --- 502.4 s compile_c --- 3830.5 s =========================================== Total: --- 7423.8 s More than half the time is spent compiling C code. The makefile uses neither precompiled headers nor parallel compilation, so there is lots of room for improvement. I tried these two optimizations, and here are the timings for the C code compilation: baseline: 3830.5 s (as above) pc: 1301.2 s pch: 404.8 s pch + pc: 165.8 s pc = parallel compilation pch = precompiled headers Using both optimizations you get a factor 23 speed up! Adding parallel compilation is easy. Adding precompiled headers is a bit more work; here I did a quick fix that is not suitable for production code, but good enough for profiling. I plan to add parallel compilation on Windows, along with some other Windows related fixes, to the PyPy code. I will not add precompiled headers now. Armin, Maciej: Does that sound OK? ---------------------------------------------------------- The main gain when using precompiled headers comes from the header forwarddecl.h. This header is 17.8 MB and is included in 635 source files. The amount of C code that is compiled is about 424 MB. But because of forwarddecl.h, you are in fact compiling another 635 x 17.8 MB = 11 GB. Compiling 424 MB or 11 GB of C code - that does make a difference. But you can not just create a precompiled header that includes forwarddecl.h and apply it to all source files. You should only apply the precompiled header to files that include forwarddecl.h to begin with. These seem to be precisely the source files with names that match one of the patterns data_*.c, nonfuncnodes*.c, implement*.c, pypy_*.c and rpython_*.c. But I don't see how to write a nmake rule that only applies to such files. (nmake does not have as good pattern matching support as GNU make.) --Johan

4 7

All orders are shipped via Express Courier delivery to ensure fast and prompt delivery.
by Yvette Cross 31 Jan '14

31 Jan '14

Hello Pypy You can prefer any watch for her that matches her taste and your resources. The perfect place to buy watches as we offer best quality, excellent service, money back guarantee in case you are not satisfied and have many other strong points, such as fast delivery, helpful and caring customer service. ******************************************* About two weeks ago, I ordered two watches. A Rolex Daytona, and a Vacheron Constantin. I found your web site by chance, and decided to take a gamble and order these two watches. To my surprise and delight, my watches arrived in just under a week. The watches are amazing. We checked the Daytona against a real one, and you cant tell the difference. First class all the way. I just received my third watch, an A. Lange & Sohne!!!! The watch is fantastic!!! Thanks! Yvette Cross ******************************************* Click here ---> http://watchpros.ru

1 0

Chocolatey package for windows ?
by Stuart Axon 30 Jan '14

30 Jan '14

http://chocolatey.org/ bills itself as apt-get for windows, it's fairly convenient - any chance of adding a pypy package ? If so, then one could install with cinst pypy S++

3 3

Memory leaks in Pypy
by Ron Reiter 30 Jan '14

30 Jan '14

Something really strange is happening to me. I've stumbled upon a memory leak that happens only when answering a POST request using Twisted and Pypy. The problem is described here: http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-h… It could either be a bug in the Python code that handles the request and leaks, a but in Twisted, or in Pypy. I've been investing the problem for weeks now, with the following conclusion: * CPython does not leak with this code * Tornado with Pypy does not leak with this code * Answering GET requests instead of POST requests does not leak with this code * I tried switching GCs, collecting using gc.collect and setting PYPY_GC_MAX, those attempts did not help * This happens both on Linux and Mac Insights as to how to debug this are very welcome. Thanks, -- Ron Reiter [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter]<http://twitter.com/#!/ronreiter> [image: LinkedIn]<http://www.linkedin.com/profile/view?id=4436607&locale=en_US&trk=tab_pro> Contact me: [image: Skype] ron.reiter

3 7

Gday Mate!
by Little Girlie 29 Jan '14

29 Jan '14

1 0

cppyy and callbacks
by Alex Stewart 29 Jan '14

29 Jan '14

Hi all, So I've been working on trying to develop PyPy bindings for the Bullet physics library (http://bulletphysics.org/) using cppyy. So far, after a bit of an initial learning curve, I was able to put together a configuration that maps almost all of the standard Bullet classes and functions, and worked surprisingly well "out of the box" (with no fixup code or anything), and have even successfully ported a basic "hello world" Bullet example from one of the tutorials to Python. (I have to say, I'm pretty impressed with how smoothly it all works!) The one main issue I'm left with at this point, however, is that for any sort of real application, Bullet makes substantial use of callbacks (both in the form of some global C++ function pointers as well as some abstract "callback" classes with virtual methods intended to be subclassed and overridden). My understanding is that cppyy does not currently support any form of calling Python code from C++ (only the other way around), so this presents a bit of a problem. >From what I've read, there are currently efforts to switch cppyy to being cling-based, and that this change might also allow some of this. I was wondering (a) how far off is this likely to be? (is it a "we'll have a beta next week" sort of thing, or a "we haven't even started yet, and might never get around to it" sort of thing?) and (b) will calling Python from C++ automatically be an option as soon as the cling-based stuff is available, or is that something that would be an additional step to add (and thus take more time) after the basic cling-transition is done? I've been futzing around with a workaround option in the meantime which I think might work: 1. Wrap the C++ function pointers/virtual functions with stub code which calls a C function pointer instead, passing C++ objects as "void *" 2. Write helper C++ functions which can accept "void *" and return a result casted to the appropriate C++ type 3. Use cffi or ctypes to have the C function pointers call back into a Python wrapper function, which then calls the helper conversion functions via cppyy to convert the "void *"s back into the correct cppyy objects, and then calls the actual Python callback function/method with those as arguments. Obviously, this is kinda clunky and involves a fair bit of overhead, which I'd really like to avoid, so I'm also curious if anybody has any other suggestions for better ways to do this sort of thing? Any help would be much appreciated! --Alex

5 24

Your message on craigslist
by Little Girlie 28 Jan '14

28 Jan '14

1 0

pypy-dev January 2014