Mailman 3 May 2014 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

europython sprints?
by Martin Matusiak 01 Jul '14

01 Jul '14

Hi, I'm wondering if there will be pypy sprints at Europython this year (?) And if so, which are the principal topics (I assume stm is one of them..)? Thanks, Martin

4 7

Getting involved
by Travis Jensen 03 Jun '14

03 Jun '14

Hi, I've been following pypy for a while and have thought it would be fun to get involved. Recently, I've been wondering what python with tail call optimization would look like (I'm disgruntled by GVR's decision :), and realized this might be a good way for me to get involved in pypy. So a few questions: First, is this within the realms of (reasonable) possibility? I don't want to dive into something that is not realistically possible. Given I'm a noob in the pypy world, where would be a good place to start getting my feet wet on this? Thanks. tj

2 1

Issue #1783: 50x higher memory usage with array.array (compared to CPython) (pypy/pypy)
by Konstantin Lopuhin 31 May '14

31 May '14

New issue 1783: 50x higher memory usage with array.array (compared to CPython) https://bitbucket.org/pypy/pypy/issue/1783/50x-higher-memory-usage-with-arr… Konstantin Lopuhin: I am trying to initialize a large array with an iterator (for example, where all values are zero initially), and PyPy consumes much more memory than CPython: ``` #!bash $ python Python 2.7.6 (v2.7.6:3a1db0d2747e, Nov 10 2013, 00:42:54) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin $ /usr/bin/time -l python -c "from array import array; array('b', (0 for _ in xrange(10000000)))" 1.52 real 1.50 user 0.01 sys 14954496 maximum resident set size 0 average shared memory size 0 average unshared data size 0 average unshared stack size 4131 page reclaims 0 page faults 0 swaps 0 block input operations 0 block output operations 0 messages sent 0 messages received 0 signals received 0 voluntary context switches 7 involuntary context switches $ pypy Python 2.7.6 (394146e9bb67, May 08 2014, 16:45:59) [PyPy 2.3.0 with GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin $ /usr/bin/time -l pypy -c "from array import array; array('b', (0 for _ in xrange(10000000)))" 1.47 real 1.13 user 0.32 sys 736083968 maximum resident set size 0 average shared memory size 0 average unshared data size 0 average unshared stack size 223518 page reclaims 1 page faults 0 swaps 0 block input operations 20 block output operations 0 messages sent 0 messages received 0 signals received 0 voluntary context switches 4 involuntary context switches ```

1 0

hunting for reason of a failing test (rpython/memory/test/)
by Gregor Wegberg 29 May '14

29 May '14

Hi, I'm working on an object pinning feature for incminimark right now. There is always one failing test (depending on the platform) in rpython/memory/test/test_incminimark_gc.py which I just can't solve. I'd appreciate any form of help. Even if it is just a "I may imagine that it is connected to X and Y" :-) Basic resources: - all my changes are in the following fork & branch: https://bitbucket.org/groggi/pypy/branch/gc-incminimark-pinning - my `gc-incminimark-pinning` branch is based on PyPy's `release-2.3.x` branch - pinning should work for "simple" objects, i.e. basically RPython strings. GC cards are not supported. More tests are needed of course. - I am "gregor_w" in #pypy, so feel free to ping me The following test results I get on the three major platforms running pytest on /rpython/memory/test: - Windows (Visual Studio 2013 native tools): http://paste.pound-python.org/show/LPw4eCHE1EsfFTykQkWA/ => failing test: test_print_leak - OS X (clang): http://paste.pound-python.org/show/l85sUl56UCMDVxu1vrSS/ => failing test: test_tagged_id - Ubuntu (gcc): http://paste.pound-python.org/show/oAYctWSzF3OpqLtlBlUX/ => failing tests: test_tagged_id I have no idea why on Windows I get an other test to fail and test_tagged_id passes. However, maybe this gives you a hint. I imagine that one of the following three reasons are responsible for the failing tests: 1.) I did some address calculations wrong inside incminimark. However, I'd expect to see way more failing tests in this case. 2.) My changes to get the GC transformation working with pin()/unpin() or the translation for the same two functions is wrong, so there is some error in rpython/memory/gctransform/framework.py, rpython/memory/gctransform/transform.py, rpython/rtyper/llinterp.py or rpython/rtyper/lltypesystem/lloperation.py. On the other hand I can translate PyPy with a PyPy binary containing my changes -- so this seems also somehow impossible?! 3.) It is some side effect as a result of the following changeset (llarena and lltype modification): https://bitbucket.org/groggi/pypy/commits/84898085a09814d1ec4f6b2b0d4202c7a… I tried my best to check all three explanations and some more. However, I just can't see what is going wrong. What I observed (only on OS X right now): - changing `taggedpointers` to False solves the problem, i.e. test passes - splitting both `print` statements inside the `fn` function into six prints with only one parameter (`print a, b, c` -> `print a\n print b\n print c`) leads to an error while executing the `print x` - splitting the `print` statements again as before *and* setting `taggedpointers` to False leads to the same error as the original version of the test without modifications. After 13-15 work hours trying to solve the bug I'm just lost. I have the feeling that it is something stupid and/or simple but my lack of PyPy knowledge stops me from seeing it. Any help is very much appreciated. Cheers and thank you in advance, Gregor

1 0

Issue #1782: Test case where pypy is more than 4x slower than CPython (pypy/pypy)
by estama 29 May '14

29 May '14

New issue 1782: Test case where pypy is more than 4x slower than CPython https://bitbucket.org/pypy/pypy/issue/1782/test-case-where-pypy-is-more-tha… estama: I've found a small example where PyPy is slower (more than 4x) than CPython. The example calculates all the possible groupings of a list of values. I've attached the example's code. Timings: time python allgroups.py 20.75s user 0.04s system 99% cpu 20.820 total time pypy allgroups.py 97.31s user 1.07s system 99% cpu 1:38.55 total

1 0

Issue #1781: pip install cython interferes pip install wheezy.web (pypy/pypy)
by jackychaowang 28 May '14

28 May '14

New issue 1781: pip install cython interferes pip install wheezy.web https://bitbucket.org/pypy/pypy/issue/1781/pip-install-cython-interferes-pi… jackychaowang: This works: ``` #!bash pip install wheezy.web pip install cython ``` But once we change the order, it stop working: ``` #!bash pip install cython pip install wheezy.web src/wheezy/web/middleware/errors.c: In function ‘__Pyx_PyBytes_Join’: src/wheezy/web/middleware/errors.c:2955:1: error: expected ‘;’ before ‘}’ token } ```

1 0

Issue #1780: pip install gevent doesn't work on pypy (pypy/pypy)
by jackychaowang 28 May '14

28 May '14

New issue 1780: pip install gevent doesn't work on pypy https://bitbucket.org/pypy/pypy/issue/1780/pip-install-gevent-doesnt-work-o… jackychaowang: Build error: * gevent/callbacks.c: 8:18: error: ‘PyThreadState’ has no member named ‘curexc_type’ * gevent/callbacks.c:11:19: error: ‘PyThreadState’ has no member named ‘curexc_value’ * gevent/callbacks.c:12:23: error: ‘PyThreadState’ has no member named ‘curexc_traceback’ ``` #!bash In file included from gevent/gevent.core.c:47210:0: gevent/callbacks.c: In function ‘gevent_handle_error’: gevent/callbacks.c:8:18: error: ‘PyThreadState’ has no member named ‘curexc_type’ type = tstate->curexc_type; ^ gevent/callbacks.c:11:19: error: ‘PyThreadState’ has no member named ‘curexc_value’ value = tstate->curexc_value; ^ gevent/callbacks.c:12:23: error: ‘PyThreadState’ has no member named ‘curexc_traceback’ traceback = tstate->curexc_traceback; ^ ```

1 0

Issue #1779: im.point() 4000x slower than CPython (pypy/pypy)
by hugovk 28 May '14

28 May '14

New issue 1779: im.point() 4000x slower than CPython https://bitbucket.org/pypy/pypy/issue/1779/impoint-4000x-slower-than-cpython hugovk: The following code takes nearly 4000 times longer on PyPy 2.2.1 compared with CPython. It takes less than a second with CPython compared with over 5 minutes on PyPy: ``` #!/usr/bin/env python from PIL import Image im = Image.open("lena.ppm") im.point(list(range(256))*3) im.point(lambda x: x) im = im.convert("I") im.point(lambda x: x*1) im.point(lambda x: x+1) im.point(lambda x: x*1+1) im.point(list(range(256))*256, 'L') ``` Test code: https://github.com/hugovk/test/tree/484 ``` 2.6 0m0.098s 2.7 0m0.042s 3.2 0m0.106s 3.3 0m0.094s 3.4 0m0.069s PyPy 5m16.867s ``` PyPy time / average CPython time = 3873.6797066 Timings from: https://travis-ci.org/hugovk/test/builds/26190010 The code is a simplified form of a Pillow test. At the moment it is skipped for PyPy as it takes too long. No coverage is involved. I've not tested on PyPy 2.3.

1 0

Issue #1778: PyGILState_Ensure should not deadlock if GIL already held (pypy/pypy)
by Nick Williams 26 May '14

26 May '14

New issue 1778: PyGILState_Ensure should not deadlock if GIL already held https://bitbucket.org/pypy/pypy/issue/1778/pygilstate_ensure-should-not-dea… Nick Williams: PyGILState_Ensure is documented as "(ensuring) that the current thread is ready to call the Python C API regardless of the current state of Python, or of the global interpreter lock". However, in PyPy, code which calls PyGILState_Ensure while already holding the GIL will deadlock. This does not happen in CPython and does not seem consistent with the Python documentation.

1 0