Mailman 3 August 2016 - pypy-issue

Issue #2367: del bytearray[:n] should be optimized (pypy/pypy)
by Naoki INADA 13 Aug '16

13 Aug '16

New issue 2367: del bytearray[:n] should be optimized https://bitbucket.org/pypy/pypy/issues/2367/del-bytearray-n-should-be-optim… Naoki INADA: del bytearray[:n] is optimized from Python 3.4 ref: https://bugs.python.org/issue19087 AsyncIO uses bytearray for I/O buffers. PyPy3 compatible to CPython 3.5 should have same optimization.

1 0

Issue #2366: Make PyPy W^X Compatible (pypy/pypy)
by Edd Barrett 12 Aug '16

12 Aug '16

New issue 2366: Make PyPy W^X Compatible https://bitbucket.org/pypy/pypy/issues/2366/make-pypy-w-x-compatible Edd Barrett: W^X is a mitigation technique whereby allocated pages are either writeable or executable, but never both. By working this way it makes it much harder for an attacker who has an existing memory vulnerability (e.g. a buffer overflow or use-after-free) to craft malicious binary code and jump to it. https://en.wikipedia.org/wiki/W%5EX OpenBSD is pushing to make all package software W^X compliant, which includes PyPy. Note however that W^X benefits everyone, not just OpenBSD users. Pax/Linux users would also be able to run PyPy. The overall model in a JIT is something like this: ``` mem = alloc_writable_pages(); write_code(mem); mprotect(mem, PROT_EXEC); ``` Then every time you need to mutate the memory, you have to `mprotect` it writeable again. I have already started work on this, but don't expect it to be ready quickly -- I think it's going to be pretty hard. Thanks Responsible: vext01

1 0

Issue #2365: rinohtype much slower on PyPy3 (pypy/pypy)
by Brecht Machiels 10 Aug '16

10 Aug '16

New issue 2365: rinohtype much slower on PyPy3 https://bitbucket.org/pypy/pypy/issues/2365/rinohtype-much-slower-on-pypy3 Brecht Machiels: ``` #!bash pip install rinohtype wget http://docutils.sourceforge.net/docs/user/rst/demo.txt rm -rf demo.rtc && time rinoh demo.txt ``` CPython 3.3.6 takes about 16 seconds on my machine. PyPy3 takes about 34 seconds. (I installed PyPy through pyenv, which reports pypy3.3-5.2-alpha1, but `pypy -V` reports "PyPy 5.2.0-alpha0")

1 0

Issue #2364: gctransformer issue while trying to translate pypy to ARM (pypy/pypy)
by Dewald Pretorius 09 Aug '16

09 Aug '16

New issue 2364: gctransformer issue while trying to translate pypy to ARM https://bitbucket.org/pypy/pypy/issues/2364/gctransformer-issue-while-tryin… Dewald Pretorius: I've followed the guide from here: http://rpython.readthedocs.io/en/latest/arm.html#arm All steps were successful, up to "pypy /home/username/pypy2-v5.3.1-src/rpython/bin/rpython -O1 --platform=arm target.py" which fails as follows: root@ubuntu:/srv/chroot/precise_arm# pypy /home/username/pypy2-v5.3.1-src/rpython/bin/rpython -O1 --platform=arm target.py [translation:info] 2.7.8 (2.5.0+dfsg-2, Feb 12 2015, 05:23:41) [PyPy 2.5.0 with GCC 4.9.2] [platform:msg] Set platform with 'arm' cc=None, using cc='gcc', version='Unknown' [translation:info] Translating target as defined by target Traceback (most recent call last): File "app_main.py", line 75, in run_toplevel File "/home/username/pypy2-v5.3.1-src/rpython/bin/rpython", line 20, in <module> main() File "/home/username/pypy2-v5.3.1-src/rpython/translator/goal/translate.py", line 216, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "/home/username/pypy2-v5.3.1-src/rpython/translator/goal/translate.py", line 173, in parse_options_and_load_target set_opt_level(config, translateconfig.opt) File "/home/username/pypy2-v5.3.1-src/rpython/config/translationoption.py", line 379, in set_opt_level config.translation.gc = config.translation.gc File "/home/username/pypy2-v5.3.1-src/rpython/config/config.py", line 68, in __setattr__ self.setoption(name, value, 'user') File "/home/username/pypy2-v5.3.1-src/rpython/config/config.py", line 112, in setoption child.setoption(self, value, who) File "/home/username/pypy2-v5.3.1-src/rpython/config/config.py", line 271, in setoption homeconfig.setoption(name, reqvalue, who2) File "/home/username/pypy2-v5.3.1-src/rpython/config/config.py", line 111, in setoption 'option %s' % (value, name)) ConflictConfigError: cannot override value to ref for option gctransformer I'd really appreciate any assistance as our tests have demonstrated a seven factor speed increase in using pypy over python2.7. I have made sure that I have all the dependencies installed that I need to build pypy from source - by building pypy from source. I've also tried translating the full interpreter root@ubuntu:/home/username/pypy2-v5.3.1-src/pypy/goal# /home/username/pypy2-v5.3.1-src/pypy/goal/pypy-c /home/username/pypy2-v5.3.1-src/rpython/bin/rpython -Ojit --platform=arm --gcrootfinder=shadowstack --jit-backend=arm targetpypystandalone.py [translation:info] 2.7.10 (7e8df3df96417c16c2d55b41352ec82c9c69c978, Aug 09 2016, 00:24:37) [PyPy 5.3.1 with GCC 4.9.2] [platform:msg] Set platform with 'arm' cc=None, using cc='gcc', version='Unknown' [translation:info] Translating target as defined by targetpypystandalone [platform:execute] sb2 -t ARM gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused /tmp/usession-release-pypy2.7-v5.3.1-13/gcctest.c -o /tmp/usession-release-pypy2.7-v5.3.1-13/gcctest.o [platform:execute] sb2 -t ARM gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused /tmp/usession-release-pypy2.7-v5.3.1-13/gcctest.c -o /tmp/usession-release-pypy2.7-v5.3.1-13/gcctest.o [platform:Error] sb2: Error: Invalid target specified, aborting. Traceback (most recent call last): File "/home/username/pypy2-v5.3.1-src/rpython/bin/rpython", line 20, in <module> main() File "/home/username/pypy2-v5.3.1-src/rpython/translator/goal/translate.py", line 216, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "/home/username/pypy2-v5.3.1-src/rpython/translator/goal/translate.py", line 155, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "/home/username/pypy2-v5.3.1-src/rpython/translator/goal/translate.py", line 97, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 6, in <module> from pypy.interpreter import gateway File "/home/username/pypy2-v5.3.1-src/pypy/interpreter/gateway.py", line 19, in <module> from pypy.interpreter.eval import Code File "/home/username/pypy2-v5.3.1-src/pypy/interpreter/eval.py", line 6, in <module> from pypy.interpreter.baseobjspace import W_Root File "/home/username/pypy2-v5.3.1-src/pypy/interpreter/baseobjspace.py", line 6, in <module> from rpython.rlib.debug import make_sure_not_resized File "/home/username/pypy2-v5.3.1-src/rpython/rlib/debug.py", line 9, in <module> from rpython.rtyper.lltypesystem import rffi File "/home/username/pypy2-v5.3.1-src/rpython/rtyper/lltypesystem/rffi.py", line 540, in <module> NUMBER_TYPES = setup() File "/home/username/pypy2-v5.3.1-src/rpython/rtyper/lltypesystem/rffi.py", line 529, in setup names = populate_inttypes() File "/home/username/pypy2-v5.3.1-src/rpython/rtyper/lltypesystem/rffi.py", line 523, in populate_inttypes platform.populate_inttypes(populatelist) File "/home/username/pypy2-v5.3.1-src/rpython/rtyper/tool/rfficache.py", line 95, in populate_inttypes sizes = sizeof_c_types(names_c, **kwds) File "/home/username/pypy2-v5.3.1-src/rpython/rtyper/tool/rfficache.py", line 45, in sizeof_c_types answer = ask_gcc(question, **kwds) File "/home/username/pypy2-v5.3.1-src/rpython/rtyper/tool/rfficache.py", line 35, in ask_gcc return build_executable_cache([c_file], eci, ignore_errors=ignore_errors) File "/home/username/pypy2-v5.3.1-src/rpython/tool/gcc_cache.py", line 28, in build_executable_cache result = platform.execute(platform.compile(c_files, eci)) File "/home/username/pypy2-v5.3.1-src/rpython/translator/platform/__init__.py", line 54, in compile ofiles = self._compile_o_files(cfiles, eci, standalone) File "/home/username/pypy2-v5.3.1-src/rpython/translator/platform/__init__.py", line 76, in _compile_o_files ofiles.append(self._compile_c_file(self.cc, cfile, compile_args)) File "/home/username/pypy2-v5.3.1-src/rpython/translator/platform/posix.py", line 40, in _compile_c_file cwd=str(cfile.dirpath())) File "/home/username/pypy2-v5.3.1-src/rpython/translator/platform/arm.py", line 47, in _execute_c_compiler self._handle_error(returncode, stderr, stdout, outname) File "/home/username/pypy2-v5.3.1-src/rpython/translator/platform/__init__.py", line 152, in _handle_error raise CompilationError(stdout, stderr) CompilationError: CompilationError(err=""" sb2: Error: Invalid target specified, aborting. """) Any advise would be really appreciated.

1 0

Issue #2363: Segfault in GC: pinned object in 'objects_to_trace' (pypy/pypy)
by 9001 03 Aug '16

03 Aug '16

New issue 2363: Segfault in GC: pinned object in 'objects_to_trace' https://bitbucket.org/pypy/pypy/issues/2363/segfault-in-gc-pinned-object-in 9001: I encountered a reproducible segfault when running a particular script, both with release builds and with lldebug. ``` PyPy assertion failed at rpython_memory_gc.c:22908: in pypy_g_IncrementalMiniMarkGC_visit: pinned object in 'objects_to_trace' ``` The script and further details are provided in the attachment.

1 0

Issue #2362: Django's development server repeats reload infinitely (pypy/pypy)
by RyoYokoyama 02 Aug '16

02 Aug '16

New issue 2362: Django's development server repeats reload infinitely https://bitbucket.org/pypy/pypy/issues/2362/djangos-development-server-repe… RyoYokoyama: [Django's development server](https://docs.djangoproject.com/ja/1.9/ref/django-admin/#runserver) repeats reload infinitely. ``` #!python C:\pypyreload>pypy manage.py runserver Performing system checks... System check identified no issues (0 silenced). You have unapplied migrations; your app may not work properly until they are applied. Run 'python manage.py migrate' to apply them. August 03, 2016 - 11:44:13 Django version 1.9.8, using settings 'pypyreload.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CTRL-BREAK. Performing system checks... System check identified no issues (0 silenced). You have unapplied migrations; your app may not work properly until they are applied. Run 'python manage.py migrate' to apply them. August 03, 2016 - 11:44:16 Django version 1.9.8, using settings 'pypyreload.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CTRL-BREAK. Performing system checks... System check identified no issues (0 silenced). ... ``` I found that in django.utils.autoreload, os.stat([path to lib-python\2.7\posixpath.py]).st_ctime returned 2 different values and this file was recognized as modified. The attached simple code reproduces this problem. My Environment: * Windows 10 64bit * pypy2.7 v5.3.1

1 0

Issue #2361: TypeError: __class__ assignment: 'A' object layout differs from 'B' (pypy/pypy)
by Craig Rodrigues 01 Aug '16

01 Aug '16

New issue 2361: TypeError: __class__ assignment: 'A' object layout differs from 'B' https://bitbucket.org/pypy/pypy/issues/2361/typeerror-__class__-assignment-… Craig Rodrigues: Alex Gaynor was able to come up with this test case which passes under CPython but fails under Pypy: ``` class A(object): __slots__ = ["a"] class B(object): __slots__ = ["a"] inst = A() inst.a = 3 inst.__class__ = B ``` I was also able to reproduce this bug in the Twisted test suite. I can do this by doing the following: ``` git clone https://github.com/twisted/twisted twisted_test1 cd twisted_test1 pypy -m unittest twisted.test.test_rebuild ```

1 0

Issue #2360: Windows ctypes.CDLL : TypeError: function name must be a string or integer (pypy/pypy)
by RyoYokoyama 01 Aug '16

01 Aug '16

New issue 2360: Windows ctypes.CDLL : TypeError: function name must be a string or integer https://bitbucket.org/pypy/pypy/issues/2360/windows-ctypescdll-typeerror-fu… RyoYokoyama: On Windows, ctypes.CDLL cannot accept unicode function name. On PyPy2: ``` #!python >>>> import ctypes >>>> lib = ctypes.CDLL('sqlite3') >>>> lib[b'sqlite3_libversion']() 1642690176 >>>> lib[u'sqlite3_libversion']() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\pypy2\lib-python\2.7\ctypes\__init__.py", line 387, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) File "c:\pypy2\lib_pypy\_ctypes\function.py", line 254, in __init__ ptr = self._getfuncptr([], ctypes.c_int) File "c:\pypy2\lib_pypy\_ctypes\function.py", line 439, in _getfuncptr self._ptr = cdll.getfunc(self.name, ffi_argtypes, ffi_restype) TypeError: function name must be a string or integer ``` On CPython 2.7: ``` #!python >>> import ctypes >>> lib = ctypes.CDLL('sqlite3') >>> lib[b'sqlite3_libversion']() 1787770432 >>> lib[u'sqlite3_libversion']() 1787770432 ```

1 0

Issue #2359: VMProf does not display traces correctly (pypy/pypy)
by Richard Plangger 01 Aug '16

01 Aug '16

New issue 2359: VMProf does not display traces correctly https://bitbucket.org/pypy/pypy/issues/2359/vmprof-does-not-display-traces-… Richard Plangger: Here is a python example: https://paste.pound-python.org/show/RXatEmOq95xg4HcneO50/ And the corresponding cpu profile: http://vmprof.com/#/88a155cf8a54c7ddc23374aa4fd407b1 The issue might be in trace_from_start of tmp_callback. Needs more investigation.

1 0