Mailman 3 August 2015 - pypy-dev

buildbot summary by hg hash w/out revision number
by Matti Picus 21 Aug '15

21 Aug '15

The build summaries, like the one here http://buildbot.pypy.org/summary?category=linux-armhf gather results together by revision so that all different types of tests for a single revision line up in a single column of results. ARM is built on one machine (cross-translated since the ARM machines do not have 4GB memory) and tested on another. Since the Mercurial build step is given an revision it does hg pull -r <rev hash> hg up -r <rev hash> which seems to cause the revision number in the testing repo to get out of sync with the revision number of the building repo, which in turn prevents gathering results by rev_int:rev:hash from the tests together with the builds. In short, I committed a change to the buildbot summary creation script (with tests) on the coalesce-by-hash branch that should fix it by only coalescing test runs by hg hash. The proof would be if the link above displayed results from all 7 armhf build slaves[0][1] together in one column, without breaking navigation around the site. Could a buildbot admin please upgrade to the branch, restart the buildbot, and click around the links on the page to see if I achieved hashing Nirvana? If things go wrong, just revert to default and try to let me know what I broke. Matti [0] Do we really need all these builds every night? [1] Here are the builders, did I forget any? build-pypy-c-jit-linux-armhf-raring build-pypy-c-jit-linux-armhf-raspbian build-pypy-c-linux-armhf-raspbian jitbackendonly-own-linux-armhf jitbackendonly-own-linux-armhf-v7 pypy-c-app-level-linux-armhf-v7 pypy-c-jit-linux-armhf-v7

2 4

Don't release GIL in jit_libffi?
by Timothy Baldridge 18 Aug '15

18 Aug '15

My calls into C are pretty fast in my custom RPython VM, but jit_libffi is trying to release the GIL on every single call to a FFI function. Is there a way to disable this if I know that a function call is going to be super fast? Timothy

4 4

pypy callback function pointer become invalid
by Yicong Huang 18 Aug '15

18 Aug '15

Hi, We embeded Pypy in C++ code, and used the callback function pointer to execute Python function. And we noticed that, the callback function pointers might become invalid as the memory usage grows. Here is the piece of test code: for(int i = 0; i < LOOP; i++){ //read python code from file char* pyBuffer = readPyFile("test_Lower.py"); pypy_execute_source_ptr(pyBuffer, &ptr); callers[i] = ptr.pyPtrs[0]; } //check 1st function pointer and call python function caller = callers[0]; ret = (*((char *(*)(char *))caller))("A"); printf("%s\n", ret); In the experiment, if the LOOP >150, caller becomes invalid, the callback to python function could not succed, and "Segmentation fault" error was reported. For more complex Python function, the valid LOOP might be smaller. I am wondering whether the error was caused by GC in Pypy that recycled the function pointers. In python fill_api() code, I have used a gloabl list to append function pointer, but it did not help. #global list no_gc = [] def fill_api(ptr): api = ffi.cast("struct pypyAPI*", ptr) api.pyPtrs[0] = test_Lower no_gc.append(api) no_gc.append(test_Lower) return api And are there any methods to prevent such errors?

4 11

pypy platform specific tests
by Vaibhav Sood 17 Aug '15

17 Aug '15

Hi, I am just starting off with pypy, I tried building pypy and running some tests from the test suite earlier and had a query regarding the tests. I was wondering if there are any tests in the test suite which help to test the built binary? I ask this because I believe there is platform specific code in the binary and if so are there are any platform specific tests (I am assuming they will be .c files and not .py files) in the code? Thanks! Vaibhav DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

2 3

arm testing buildbot builders
by Matti Picus 17 Aug '15

17 Aug '15

The TranslatedTesting target was not copying the cffi import libraries in nightly runs. I committed an (untested, sorry) change, 3275430ea6da, to the buildbot repo, could someone merge that to the build master? BTW, it turns out that the hg pypy repo for pypy-c-app-level-linux-armhf-v7 and pypy-c-jit-linux-armhf-v7 somehow got the version numbers out of sync with upstream, which caused the test results to be ignored on the summary page. I hope I have fixed that too. Matti

2 1

Re: [pypy-dev] do not raise exceptions in __init__
by Armin Rigo 16 Aug '15

16 Aug '15

Hi Matti, Replying to commit "614e585": "do not raise exceptions in __init__" The __init__ can raise exceptions, it's not the problem here. This commit changes (only) something else: it moves around which "*" turn into an overflow-checking multiplication, and which ones don't. The "*" operation in RPython, like the other common arithmetic operations, turn by default into C code as a "*" that assumes that overflow does not occur (and the C compiler sometimes optimizes based on this). On the other hand, if the "*" operation in RPython appears immediately inside a "try: except OverflowError:" (and only in this case), then in C it will also do the same check. Notably, this checkin fixes this code (by removing it): try: length = support.product(shape) except OverflowError as e: ... which after translation to C would never catch an OverflowError occurring implicitly inside product(). A bientôt, Armin.

2 2

weird armhf-v7 errors
by Maciej Fijalkowski 16 Aug '15

16 Aug '15

Hi I wonder why this: http://buildbot.pypy.org/builders/pypy-c-jit-linux-armhf-v7/builds/745/step… has errors that don't show up in the summary anywhere? does anyone know? Cheers, fijal

2 2

memory control in Python (fwd)
by Laura Creighton 15 Aug '15

15 Aug '15

This showed up on python-list. https://mail.python.org/pipermail/python-list/2015-August/695441.html (if you want to read his attatchments, but, grrr you need libreoffice to read them). Cursory googling for CPLEX indicates it's written in C++. Do we have any joy for this man? Laura ------- Forwarded Message Return-Path: <python-list-bounces+lac=openend.se(a)python.org> Received: from mail.python.org (mail.python.org [82.94.164.166]) Subject: memory control in Python From: Ping Liu <yanzhipingliu(a)gmail.com> To: python-list(a)python.org Dear All, I am working on an optimization problem, where we are trying to minimize some indicators like energy usage, energy cost, CO2 emission. In this problem, we have a bunch of energy conversion technologies for electricity and thermal purpose, such as heat pump, boiler, chiller, etc.. We are trying to model it with a one year time period. the current time step is one hour. We have preselected the candidate technologies to exclude those we don't want to study so that the problem size could be reduced with a limited candidate technologies. In the current case study, we only analyze the electric chiller and heat pump to supply the cooling load, while power grid will supply the electricity for all electric loads. There are binary variables regarding installation decisions of technologies and continuous variables related to installation capacity and hourly operational decisions. For small cases, Python works well. But if we consider longer time period. then it would fail due to the memory usage issues. We have tested several case studies to check the memory use for different time period, including 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours each day, as well as 4) 30 days with 24 hours each day. The first 3 cases are feasible while the last case gives out the memory error. When we are testing the forth case, the memory error comes out while creating the inequality constraints. The problem size is 1) Aeq: 12 * 26, Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq: 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error. The solver is CPLEX (academic). It turns out that the solver is taking a lot of memory as you can see in the memory test report. for the first three cases, different memory usage is observed, and it grows up dramatically with the increase of the time period. 1) solver memory usage: 25.6 MB, 2) 19.5 MB; 3) solver memory usage: 830.0742 MB. Based on my observations, I have some of the following questions regarding 1) In order to create the optimization problem (Aeq, Aineq), how can we reduce the memory usage in python programming? 2) how can I reduce the memory usage of different solvers in Python? Why would the memory decrease for the CPLEX solver within the 24-hours-in-one-day case compared with the case 1? Thanks in advance, Ping

2 1

[newb] failed build 2.6.0 on fedora 22
by Neal Becker 14 Aug '15

14 Aug '15

I thought I'd try grabbing the fedora 2.4.0 srpm and see if it would build 2.6.0, but didn't get far: PYPY_USESSION_DIR=/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal PYPY_USESSION_BASENAME=pypy python ../../rpython/bin/rpython --output=pypy --gcrootfinder=shadowstack -Ojit targetpypystandalone [translation:info] 2.7.10 (default, Jul 5 2015, 14:15:43) [translation:info] [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] [platform:msg] Set platform with 'host' cc=None, using cc='gcc', version='Unknown' [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno- unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/pypy/goal/usession-pypy-1/gcctest.o [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt - o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/gcctest [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno- unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/pypy/goal/usession-pypy-1/gcctest.o [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt - o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/gcctest [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno- unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/platcheck_0.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/pypy/goal/usession-pypy-1/platcheck_0.o [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno- unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/rpython/translator/c/src/thread.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/rpython/translator/c/src/thread.o [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/pypy/goal/usession-pypy-1/platcheck_0.o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/rpython/translator/c/src/thread.o -pthread -Wl,--export-dynamic -lrt -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession- pypy-1/platcheck_0 [translation:info] Error: [translation:info] File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/rpython/translator/goal/translate.py", line 284, in main [translation:info] default_goal='compile') [translation:info] File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/rpython/translator/driver.py", line 562, in from_targetspec [translation:info] spec = target(driver, args) [translation:info] File "targetpypystandalone.py", line 244, in target [translation:info] raise Exception("Cannot use the --output option with PyPy " [translation:ERROR] Exception: Cannot use the --output option with PyPy when --shared is on (it is by default). See issue #1971. [translation] start debugger... > /home/nbecker/rpmbuild/BUILD/pypy-2.6.0- src/pypy/goal/targetpypystandalone.py(244)target() -> raise Exception("Cannot use the --output option with PyPy "

4 3

is the handling of python signals a language implementation detail?
by Laura Creighton 10 Aug '15

10 Aug '15

I am reading 18.8.1.1 and 18.8.1.2 in https://docs.python.org/3/library/signal.html which clearly is written from the CPython point of view. ....A Python signal handler does not get executed inside the low-level (C) signal handler. Instead, the low-level signal handler sets a flag which tells the virtual machine to execute the corresponding Python signal handler at a later point(for example at the next bytecode instruction) ... Python signal handlers are always executed in the main Python thread, even if the signal was received in another thread ... But is this part of the language definition? Or an implementation detail? Laura

3 2