Mailman 3 April 2018 - pypy-dev

Problem with PyPy 5.10 and persistent module (ZODB)
by Etienne Robillard 10 Apr '18

10 Apr '18

Hello, I'm trying to run ZODB 5.3.0 with PyPy 5.10 but getting an unexpected error: erob@projectstreetwise:/home/www/projectstreetwise.org/trunk$ bin/runzeo.sh Traceback (most recent call last): File "/usr/local/pypy/bin/runzeo", line 11, in <module> load_entry_point('ZEO==5.1.0', 'console_scripts', 'runzeo')() File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 390, in main options.realize(args) File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 127, in realize ZDOptions.realize(self, *a, **k) File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/zdaemon-4.2.0-py2.7.egg/zdaemon/zdoptions.py", line 259, in realize arg = handler(arg) File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 74, in handle_filename from ZODB.config import FileStorage # That's a FileStorage *opener*! File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/ZODB-5.3.0-py2.7.egg/ZODB/__init__.py", line 17, in <module> from persistent import TimeStamp File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/persistent/__init__.py", line 30, in <module> from persistent.persistence import Persistent File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/persistent/persistence.py", line 30, in <module> from . import ring File "/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/persistent/ring.py", line 161, in <module> """, include_dirs=[this_dir]) File "/usr/local/pypy/pypy2-v5.10.0-linux64/lib_pypy/cffi/api.py", line 444, in verify lib = self.verifier.load_library() File "/usr/local/pypy/pypy2-v5.10.0-linux64/lib_pypy/cffi/verifier.py", line 103, in load_library self._write_source() File "/usr/local/pypy/pypy2-v5.10.0-linux64/lib_pypy/cffi/verifier.py", line 192, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/usr/local/pypy/pypy2-v5.10.0-linux64/site-packages/persistent/__pycache__/_cffi__g2deb1cb8xa682c2c7.c' Any ideas how to fix this problem? Thank you in advance. Etienne

1 1

PyPy threads
by Rene Nejsum 08 Apr '18

08 Apr '18

I have been running some test on PyPy3 1) hope someone can confirm/explain this: If PyPy is running a tight loop in a thread, other threads will not be run/scheduled? The example: I am running factorial(200000) in a thread: 2) class MathActor(Actor): def factorial(self, n): logger.debug("factorial(%d)" % n) i = n while i > 1: i -= 1 n *= i logger.debug("fact done...") return n On pypy3 all other threads (100+) seams to stop/block, while factorial() is executing, the log: 180404 142203.094 INFO [worker:3:WorkerTask] timeout(): 4 180404 142203.094 INFO [worker:3:WorkerTask] Starting math job *180404 142203.095 DEBUG factorial(200000)* *180404 142234.486 DEBUG fact done…* 180404 142234.487 INFO [netclient:9:EchoClientTask] Client: timeout: 127.0.0.1:5050 180404 142234.488 INFO [worker:3:WorkerTask] timeout(): 5 180404 142238.257 INFO [worker:3:WorkerTask] result from factorial, has 973351 digits I suspect the JIT sits in a very tight loop? But the OS should schedule other threads? Event though the don’t do I/O? Most of them doing queue.get(timeout=5) waiting for new messages. I did a quick google, but could not find anything about this on pypy3. On cpython3 3) the other threads seams to run while the factorial thread is running 180404 142502.124 INFO [netclient:9:EchoClientTask] Client received: 'From Server count: 4' 180404 142506.494 INFO [worker:3:WorkerTask] timeout(): 4 180404 142506.494 INFO [worker:3:WorkerTask] Starting math job *180404 142506.495 DEBUG factorial(200000)* 180404 142507.135 INFO [netclient:9:EchoClientTask] Client: timeout: 127.0.0.1:5050 180404 142507.368 INFO [netclient:9:EchoClientTask] Client received: 'From Server count: 5' 180404 142511.506 INFO [worker:3:WorkerTask] timeout(): 5 180404 142511.519 INFO [worker:3:WorkerTask] setting value for long_work().... 180404 142511.540 INFO [client3:6:ClientTask] Wait: got 3 = 42 180404 142511.546 INFO [client3:6:ClientTask] Worker Task done: Good work... 180404 142512.386 INFO [netclient:9:EchoClientTask] Client: timeout: 127.0.0.1:5050 180404 142512.598 INFO [netclient:9:EchoClientTask] Client received: 'From Server count: 6' 180404 142516.538 INFO [worker:3:WorkerTask] timeout(): 6 180404 142516.548 INFO [client3:6:ClientTask] Worker Task done: Good work... 180404 142517.616 INFO [netclient:9:EchoClientTask] Client: timeout: 127.0.0.1:5050 180404 142517.776 INFO [netclient:9:EchoClientTask] Client received: 'From Server count: 7' *180404 142520.666 DEBUG fact done…* 180404 142521.554 INFO [worker:3:WorkerTask] timeout(): 7 180404 142535.265 INFO [netclient:9:EchoClientTask] Client: timeout: 127.0.0.1:5050 180404 142535.268 INFO [worker:3:WorkerTask] result from factorial, has 973351 digits br /Rene 1) pypy3 installed with brew Python 3.5.3 (3f6eaa010fce78cc7973bdc1dfdb95970f08fed2, Jan 13 2018, 18:14:01) [PyPy 5.10.1 with GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] 2) https://github.com/pylots/pyworks.git <https://github.com/pylots/pyworks.git> 3) python3 Python 3.6.5 (default, Mar 30 2018, 06:41:53) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin 4) HW MacBook (Retina, 12-inch, 2017) 1,4 GHz Intel Core i7 (dual core)

1 0

Progress - I have run and uploaded PyPy benchmarks
by Matti Picus 02 Apr '18

02 Apr '18

I have run and uploaded PyPy benchmarks. I am not to sure I have done it correctly, here is what I did https://gist.github.com/mattip/e99d5a3873df4460daffba0b74c8e23e\ For some reason, I needed to go to the admin site https://speed.python.org/admin/codespeed/report/ and add a report for the run. The results show up on https://speed.python.org/comparison/, better viewed in horizontal mode (should this be the default?) and by reducing the number of executables. They do not show up on the home page, changes, nor timeline tabs. Any suggestions? Matti

1 0

Sqlite tests broken for pypyp 3.6
by Jean-Daniel 02 Apr '18

02 Apr '18

Hello team, Here is Arianna and I's understanding of the problem we see on the buildbot [1]. 1. The tests are failing because CPython's sqlite3 module changed between 3.5 and 3.6 and Pypy's sqlite3 module needs to be modified to reflect that. 2. CPython's s sqlite3 now "no longer implicitly commits an open transaction" [2] and we ported that commit in Pypy 3. Now, the "CheckTypeMapUsage" test [3] started to break, but that is because that test relies on CPython's deterministic garbage collector: 4. The test creates a cursor that does a SELECT on a table. The SELECT cursor does not go through all the data, before a DROP is executed on the table. The sqlite3 library keeps track of the unfinalized SELECT and usually rejects such a DROP with a 'database locked' error message. This is to make sure the SELECT does silently get an incomplete view. 5. In CPython, the test does pass, because the SELECT cursor is not assigned to a variable, and is deleted right away by the GC, which finalizes the cursor, which unlocks the database. 6. We checked that, when the test is modified so that the cursor gets assigned to a variable, then CPython's test receives the 'database locked' error when the DROP is attempted. This confirms the point above. 7. In Pypy, the test used to pass in 3.5 because of a workaround implemented in Pypy's sqlite3 module [4] that relies on "open transactions being automatically committed", behavior which got removed in [2]. Our understanding is that the workaround breaks a guard from Sqlite, also it introduces a significant skew between CPython and Pypy. Here are the steps we suggest: a. fix the CheckTypeMapUsage test that relies on the GC behavior in CPython b. update the tests in Pypy c. remove the workaround in Pypy d. port the commit in [2] from CPython to Pypy. What do you think about these steps? Thanks a lot for your warm welcome at the Pypy Sprint, very enjoyable :) [1]: http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy -c-app-level-linux-x86-64&build=3945&mod=lib-python.3.test.test_sqlite [2]: https://github.com/python/cpython/commit/ab994ed8b97e1b0dac151ec827c857 f5e7277565 [3]: https://github.com/python/cpython/blob/55966f3a0d5f1bf823bd22ce1abbde267b06… [4]: https://bitbucket.org/pypy/pypy/src/045483a22d662e8f5305ef5c8b144623a2ff8c5…

1 0

sqlite3 tests broken for python 3.6
by Jean-Daniel 02 Apr '18

02 Apr '18

Hello team, Here is Arianna and I's understanding of the problem we see on the buildbot [1] 1.the tests are failing because CPython's sqlite3 module changed between 3.5 and 3.6 and the tests were modified to reflect that. 2. CPython's s [1]: http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy-… [2]: https://github.com/python/cpython/commit/ab994ed8b97e1b0dac151ec827c857f5e7…

1 0