Mailman 3 April 2014 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

Virtualizables in RPython
by Anton Gulenko 30 Apr '14

30 Apr '14

Dear developers, I am currently working on my Masters thesis about optimizations for SPy (Squeak VM written using the RPython toolchain). I am planning to devote one or two chapters to the issue of virtualizable objects in a VM like SPy (and also in SPy in particular). Since there is not much documentaion on virtualizables available, I would appreciate your input. I want to collect details about the underlying concept, and also about the specific implementation in the RPython JIT. For example, was this concept first introduced in Pypy, or is it an older idea? How exactly does the optimizer decide which objects can be virtualized, and which can not? I would also appreciate pointers to relevant parts of the Pypy source code, which is probably the best documentation as of now. Thank you and best regards, Anton

4 7

Another benchmark where Pypy is much slower than CPython
by Robert Grosse 29 Apr '14

29 Apr '14

Hi, while working on some MDP code, I discovered that it was several times as slow under Pypy as CPython. Attached is a benchmark demonstrating this. On my computer, the benchmark takes 70.3 seconds under CPython 2.7.3 64bit, but it takes 273.6 seconds under pypy-c-jit-71056-c8e3b8cbc843-win32, the latest version I could find. The code is not at all optimized, but I still found it strange that Pypy did so much worse. The code doesn't seem that complex, but perhaps the usage of nameduples is triggering pathological behavior from the JIT's point of view.

2 2

release of pypy2.3 is imminent
by Matti Picus 27 Apr '14

27 Apr '14

If you have a branch you would like included in pypy-2.3.0, please be sure to merge it by April 28. At that point I will merge, for the last time, default into the release branch. Matti

4 3

Interesting blog post
by Laura Creighton 26 Apr '14

26 Apr '14

https://stripe.com/blog I wrote them, and yes non-USA citizens/green card holders are eligible. So if somebody wants to get paid to live in San Francisco for 3 months and hack on PyPy, send in an application .... Laura

2 1

pickle/cPickle bug
by Martin Koch 23 Apr '14

23 Apr '14

Hi list I have found what appears to be a bug in the pickle and cPickle library in pypy. I can reproduce it both on linux and mac. Basically, I can't unpickle a file if it has been pickled with a file object returned by open(), but it works if I use the with construct. The problem is present both using pickle and cPickle. On mac, the pypy version is Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 20:57:21) [PyPy 2.2.1 with GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] On linux, the pypy version is Python 2.7.3 (2.2.1+dfsg-1, Jan 24 2014, 10:12:37) [PyPy 2.2.1 with GCC 4.6.3] *#import cPickle as pickle* *import pickle* *l = range(10)* *# fails:* *try:* * pickle.dump(l, open('f', 'wb'))* * print pickle.load(open('f', 'rb'))* *except EOFError:* * print("Yup, that's an EOFError")* *#succeeds* *with open('f', 'wb') as f:* * pickle.dump(l, f)* *with open('f', 'rb') as f:* * print pickle.load(f)* If I instead of using open() inline in the argument to pickle assign it to a variable and explicitly close the file after calling pickle.dump, then the problem goes away: * f = open('f', 'wb')* * pickle.dump(l, f)* * f.close()* * print pickle.load(open('f', 'rb'))* Apparently, in the first code snippet, the file isn't closed as it should be when the object returned by open() goes out of scope after pickle.dump. Thanks, /Martin Koch

2 1

2.3 release is close, please help triage the open bugs
by Matti Picus 23 Apr '14

23 Apr '14

The consensus on IRC is that we are ready to release PyPy 2.3 Please help triage the bugs on bugs.pypy.org, and mark your favorite release-critical one with the 2.3 Release tag. We could also use a name for the release, suggestions are welcome. For an idea as to what changed, see the release notice https://bitbucket.org/pypy/pypy/src/20e51c4389ed4469b66bb9d6289ce0ecfc82c4b… Matti

2 1

Benchmark that pypy doesn't do well on
by Kevin Modzelewski 23 Apr '14

23 Apr '14

Hi all, I've finally gotten around to open-sourcing some old code of mine (coincidentally, the predecessor to Pyston) which doesn't perform that well on PyPy. It's not the best benchmark, since it's not deterministic and produces different results on PyPy and CPython (due to dict ordering, I think), but PyPy seems to be consistently 20-30% slower than CPython, so I think the overall effect is reliable even if the exact numbers aren't meaningful. The benchmark is "real code" in that I simply added a benchmark mode to a real project, where the benchmark runs the exact same thing as a normal invocation (except for disabling some disk output); I'm definitely not trying to claim that the benchmark is "representative" in any sense, though. To run it: cd ~ # needs to be in the home directly unfortunately git clone https://github.com/kmod/icbd # set up python env time bash icbd/icbd/type_analyzer/run.sh bench On my machine, CPython runs it in 60s, but PyPy takes 75s; this is on PyPy 1.8 since I can't get a newer version, but I get similar results with 2.2.1. I should mention that the program doesn't use any non-stdlib modules, at least in "bench" mode. I've also tried to extract a part of the program that seemed to run significantly slower under PyPy, and got this microbenchmark: https://github.com/dropbox/pyston/blob/master/microbenchmarks/polymorphism.… It takes about 13s for PyPy, and about 4s for CPython. This microbenchmark is only based on the final phase of the program (the part right before printing all the colored output), so I don't think it's necessarily that representative, it's just something I happened to notice was much slower. Just wanted to send it over in case you guys were interested. kmod

2 1

Blocked Block error
by Ryan Gonzalez 23 Apr '14

23 Apr '14

First, a quick apology: I don't want anyone to feel like I'm abusing the list or something. I just really need help, and StackOverflow isn't necessarily RPython-friendly. What I want to know is the meaning of the Blocked Block error. I always get that error at least once. Usually I can solve it myself, but I'm curious as to what it actually means. -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated."

2 2

Alternatives to multiple inheritance
by Ryan Gonzalez 19 Apr '14

19 Apr '14

I’m writing a scripting language using RPython. I have a base class for all my objects. Now, I have an exception object. The exception object needs to derive from my base class in order for me to use polymorphism inside the interpreter. However, it also needs to derive from the Exception class to be throwable. My code looked kind of like this: class BaseObject(object): pass class MyException(BaseObject, Exception): ... However, upon trying to compile the code, I got this error: [translation:ERROR] AssertionError: multiple inheritance only supported with _mixin_: <class 'bolt.objects.BoltException'> [translation:ERROR] Processing block: [translation:ERROR] block@122 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> [translation:ERROR] in (bolt.main:10)parse_and_run [translation:ERROR] containing the following operations: [translation:ERROR] v0 = simple_call((type interpreter)) [translation:ERROR] --end-- bolt.objects.BoltException is my exception class. If I remove the Exception base, I get this: [translation:ERROR] AssertionError [translation:ERROR] Processing block: [translation:ERROR] block@9 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> [translation:ERROR] in (bolt.main:4)run_file [translation:ERROR] containing the following operations: [translation:ERROR] v0 = simple_call((function create_file), p_0, ('r')) [translation:ERROR] v1 = getattr(v0, ('read')) [translation:ERROR] v2 = simple_call(v1) [translation:ERROR] v3 = simple_call((function parse_and_run), v2) [translation:ERROR] v4 = getattr(v0, ('close')) [translation:ERROR] v5 = simple_call(v4) [translation:ERROR] --end-- The only thing about multiple inheritance and RPython I found was this<https://mail.python.org/pipermail//pypy-commit/2011-November/055925.html> . My current idea is to have my base object derive from Exception instead of nothing. I was wondering if there’s a better solution, however. -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated."

3 3