Mailman 3 May 2014 - pypy-issue

[issue1770] Massive performance regression in fastavro with PyPy >= 1.9
by Armin Rigo 15 May '14

15 May '14

Armin Rigo <armin.rigo(a)gmail.com> added the comment: Massive slowdown in the fastavro library between PyPy 1.8 and 1.9. (This message is here to send a mail to the mailing list; the original one is too big with its attached file.) ---------- nosy: +arigo status: unread -> chatting ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1770> ________________________________________

1 0

[issue1767] stm - KeyError in JIT's bridge optimization
by Hubert Hesse 15 May '14

15 May '14

New submission from Hubert Hesse <hubert.hesse(a)student.hpi.uni-potsdam.de>: Hello, we are building an stm-enabled lang-smalltalk (see bitbucket's stmgc-c7 branch in lang-smalltalk) with the pypy commit 4169f1423e3d (on pypy's stmgc-c7 branch). We have encountered a KeyError in the bridge optimization of the JIT in several parts of the code. We assume a common root cause. First, we encountered this error: Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". ##################RPython traceback: File "spyvm_interpreter.c", line 22899, in Interpreter_c_loop_stm File "rpython_jit_metainterp_warmstate.c", line 499, in maybe_compile_and_run__star_4 File "rpython_jit_metainterp_warmstate.c", line 830, in execute_assembler__star_1 File "rpython_jit_metainterp_compile.c", line 11956, in ResumeGuardDescr_handle_fail File "rpython_jit_metainterp_compile.c", line 12646, in ResumeGuardDescr__trace_and_compile_from_bridge File "rpython_jit_metainterp_pyjitpl.c", line 55090, in MetaInterp_handle_guard_failure File "rpython_jit_metainterp_pyjitpl.c", line 55606, in MetaInterp__handle_guard_failure File "rpython_jit_metainterp_pyjitpl.c", line 1475, in MetaInterp_interpret File "rpython_jit_metainterp_pyjitpl.c", line 2516, in MetaInterp__interpret File "rpython_jit_metainterp_pyjitpl.c", line 3686, in MIFrame_run_one_step File "rpython_jit_metainterp_pyjitpl.c", line 26298, in MIFrame_opimpl_jit_merge_point File "rpython_jit_metainterp_pyjitpl.c", line 38787, in MetaInterp_reached_loop_header File "rpython_jit_metainterp_pyjitpl.c", line 48282, in MetaInterp_compile_loop File "rpython_jit_metainterp_compile.c", line 7021, in compile_retrace File "rpython_jit_metainterp_optimizeopt___init__.c", line 196, in optimize_trace File "rpython_jit_metainterp_optimizeopt_unroll.c", line 11521, in UnrollOptimizer_close_loop File "rpython_jit_metainterp_optimizeopt_unroll.c", line 13907, in UnrollOptimizer__import_op File "rpython_jit_metainterp_optimizeopt_unroll.c", line 14971, in UnrollOptimizer_import_box File "rpython_rtyper_lltypesystem_rdict.c", line 23396, in ll_dict_getitem__dicttablePtr_objectPtr_2 ~~~ Crash in JIT! <KeyError object at 0x61c02e0> then we applied the following changes: --- a/rpython/jit/metainterp/optimizeopt/unroll.py Wed May 07 14:33:16 2014 +0200 +++ b/rpython/jit/metainterp/optimizeopt/unroll.py Mon May 12 13:43:43 2014 +0200 @@ -481,6 +481,8 @@ if box in self.boxes_created_this_iteration: return + if not self.short_boxes.has_producer(box): + return short_op = self.short_boxes.producer(box) newresult = self.add_op_to_short(short_op) Next we saw another error through the bridge optimization: ###########################RPython traceback: File "spyvm_interpreter.c", line 22892, in Interpreter_c_loop_stm File "rpython_jit_metainterp_warmstate.c", line 499, in maybe_compile_and_run__star_4 File "rpython_jit_metainterp_warmstate.c", line 830, in execute_assembler__star_1 File "rpython_jit_metainterp_compile.c", line 11951, in ResumeGuardDescr_handle_fail File "rpython_jit_metainterp_compile.c", line 12641, in ResumeGuardDescr__trace_and_compile_from_bridge File "rpython_jit_metainterp_pyjitpl.c", line 55085, in MetaInterp_handle_guard_failure File "rpython_jit_metainterp_pyjitpl.c", line 55601, in MetaInterp__handle_guard_failure File "rpython_jit_metainterp_pyjitpl.c", line 1475, in MetaInterp_interpret File "rpython_jit_metainterp_pyjitpl.c", line 2517, in MetaInterp__interpret File "rpython_jit_metainterp_pyjitpl.c", line 3687, in MIFrame_run_one_step File "rpython_jit_metainterp_pyjitpl.c", line 26295, in MIFrame_opimpl_jit_merge_point File "rpython_jit_metainterp_pyjitpl.c", line 38783, in MetaInterp_reached_loop_header File "rpython_jit_metainterp_pyjitpl.c", line 48008, in MetaInterp_compile_loop File "rpython_jit_metainterp_compile.c", line 3247, in compile_loop File "rpython_jit_metainterp_inliner.c", line 1004, in Inliner_inline_op File "rpython_jit_metainterp_inliner.c", line 1821, in Inliner_inline_snapshot File "rpython_jit_metainterp_inliner.c", line 1733, in Inliner_inline_snapshot File "rpython_rtyper_lltypesystem_rdict.c", line 23438, in ll_dict_getitem__dicttablePtr_objectPtr_2 ~~~ Crash in JIT! <KeyError object at 0x61c02e0> After this we disabled the inliner: @@ -154,5 +155,4 @@ part.operations = [part.operations[-1]] + \ - [inliner.inline_op(h_ops[i]) for i in range(start, len(h_ops))] + \ [ResOperation(rop.JUMP, [inliner.inline_arg(a) for a in jumpargs], None, descr=jitcell_token)] target_token = part.operations[0].getdescr() ... we encountered a similar error in the compile loop: ###########################RPython traceback: File "spyvm_interpreter.c", line 3638, in portal File "rpython_jit_metainterp_warmstate.c", line 619, in maybe_compile_and_run__star_4 File "rpython_jit_metainterp_warmstate.c", line 1004, in bound_reached__star_4 File "rpython_jit_metainterp_pyjitpl.c", line 532, in compile_and_run_once___rpython_jit_metainterp_ji File "rpython_jit_metainterp_pyjitpl.c", line 1004, in MetaInterp__compile_and_run_once File "rpython_jit_metainterp_pyjitpl.c", line 1475, in MetaInterp_interpret File "rpython_jit_metainterp_pyjitpl.c", line 2516, in MetaInterp__interpret File "rpython_jit_metainterp_pyjitpl.c", line 3687, in MIFrame_run_one_step File "rpython_jit_metainterp_pyjitpl.c", line 26293, in MIFrame_opimpl_jit_merge_point File "rpython_jit_metainterp_pyjitpl.c", line 38781, in MetaInterp_reached_loop_header File "rpython_jit_metainterp_pyjitpl.c", line 48005, in MetaInterp_compile_loop File "rpython_jit_metainterp_compile.c", line 3255, in compile_loop File "rpython_rtyper_lltypesystem_rdict.c", line 23476, in ll_dict_getitem__dicttablePtr_objectPtr_2 ~~~ Crash in JIT! <KeyError object at 0x61c02e0> ---------- messages: 6884 nosy: hubx, pypy-issue priority: bug status: unread title: stm - KeyError in JIT's bridge optimization ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1767> ________________________________________

2 4

[issue1768] Intermittent segfaults with Coverage.py
by Hugo 15 May '14

15 May '14

New submission from Hugo <hugovk+pypy(a)gmail.com>: Pillow (the 'friendly' PIL fork) is tested on all Python versions available on Travis CI: Python 2.6, 2.7, 3.2, 3.3, 3.4 and PyPy. Since adding coverage using Coverage.py, we've been seeing intermittent segfaults on the PyPy build. Sometimes all tests will pass, sometimes one will fail, sometimes many will fail. You often get a different result re-running the same build. For example: https://travis-ci.org/python-imaging/Pillow/jobs/25064994 ``` running test_file_psd ... === error 35584 Segmentation fault running test_file_tar ... === error 35584 Segmentation fault running test_file_tiff ... === error 35584 Segmentation fault ``` A build with coverage enabled fails intermittently: https://travis-ci.org/hugovk/Pillow/builds/25073481 A build with coverage removed passes: https://travis-ci.org/hugovk/Pillow/builds/25073451 I re-ran it a few times with similar results. Also, recording coverage makes the PyPy build job much slower, about x5.8 slower. Coverage doesn't affect the Python 2.x or 3.x times much. Covered build total: 28m38s Non-covered build total: 21m22s - Covered PyPy job: 10m39s Non-covered PyPy job: 3m50s - Covered Python 2.x and 3.x jobs (five jobs): 17m59s Non-covered Python 2.x and 3.x jobs (five jobs): 17m32s - Covered selftest.py: 7.624s Non-covered selftest.py: 2.860s - Covered Tests/run.py: 7m29.489s Non-covered Tests/run.py: 1m17.394s Travis CI says it's using PyPy 2.2.1: ``` python --version Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 18:48:13) [PyPy 2.2.1 with GCC 4.6.3] ``` (Here's the issue at Pillow: https://github.com/python-imaging/Pillow/issues/640) ---------- messages: 6894 nosy: hugovk, pypy-issue priority: bug release: 2.2 status: unread title: Intermittent segfaults with Coverage.py ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1768> ________________________________________

2 2

[issue1769] Memory leak
by bug 14 May '14

14 May '14

New submission from bug <ddvmssr+2pxkfw(a)sharklasers.com>: This code leaks memory on PyPy 2.3 but not CPython http://pastebin.com/zwdPgi2n ---------- messages: 6895 nosy: bug, pypy-issue priority: bug status: unread title: Memory leak ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1769> ________________________________________

1 3

[issue725] Bug in zipimporting in nested packages
by mattip 14 May '14

14 May '14

mattip <matti.picus(a)gmail.com> added the comment: It seems fixed, and I couldn't find an appropriate failing test on latest win32 buildbots pypy-c-jit-71467-67067c56ae1a-win32\pypy Python 2.7.6 (67067c56ae1a, May 11 2014, 22:01:49) [PyPy 2.4.0-alpha0 with MSC v.1500 32 bit] on win32 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``all software improvements happen by a bit'' >>>> import sys >>>> sys.path.append(r'<path>\Downloads\0.zip') >>>> import my_package >>>> my_package <module 'my_package' from '<path>\\Downloads\\0.zip\\my_package\\__init__.py'> ---------- nosy: +mattip status: chatting -> testing ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue725> ________________________________________

1 0

[issue833] missing functions in the os module
by teepark 13 May '14

13 May '14

New submission from teepark <travis.parker(a)gmail.com>: I ran into this when I needed os.fchmod, but there are quite a few names not present in pypy's os module (I'm on x86_64 linux, and I'm sure many of these aren't guaranteed to be there for any platform, but pypy on x86_64 linux should probably include them): - getlogin - confstr - getresgid - tcsetpgrp - statvfs - setresgid - initgroups - fstatvfs - getresuid - fchmod - setresuid - tcgetpgrp - confstr_names - setgroups - pathconf - fchown - openpty - ctermid - statvfs_result Please change the priority if "feature" isn't appropriate - it doesn't feel right but I don't know that cpython compatibility issues are being treated as bugs. ---------- messages: 2975 nosy: pypy-issue, teepark priority: feature status: unread title: missing functions in the os module ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue833> ________________________________________

6 13

[issue1764] pypy 2.3 doesn't translate with gcc 4.9 on x86_64
by Sven-Hendrik Haase 12 May '14

12 May '14

New submission from Sven-Hendrik Haase <sh(a)lutzhaase.com>: Upon trying to translate pypy 2.3 with gcc 4.9 I get this: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "app_main.py", line 72, in run_toplevel [translation:ERROR] File "/home/svenstaro/src/pypy/rpython/translator/c/gcc/trackgcroot.py", line 2074, in <module> [translation:ERROR] tracker.process(f, g, filename=fn) [translation:ERROR] File "/home/svenstaro/src/pypy/rpython/translator/c/gcc/trackgcroot.py", line 1967, in process [translation:ERROR] tracker = parser.process_function(lines, filename) [translation:ERROR] File "/home/svenstaro/src/pypy/rpython/translator/c/gcc/trackgcroot.py", line 1482, in process_function [translation:ERROR] table = tracker.computegcmaptable(self.verbose) [translation:ERROR] File "/home/svenstaro/src/pypy/rpython/translator/c/gcc/trackgcroot.py", line 59, in computegcmaptable [translation:ERROR] self.findframesize() [translation:ERROR] File "/home/svenstaro/src/pypy/rpython/translator/c/gcc/trackgcroot.py", line 295, in findframesize [translation:ERROR] insn1.framesize = size_at_insn1 [translation:ERROR] File "/home/svenstaro/src/pypy/rpython/translator/c/gcc/../../../../rpython/translato r /c/gcc/instruction.py", line 259, in __setattr__ [translation:ERROR] "unrecognized function prologue - " [translation:ERROR] AssertionError: unrecognized function prologue - only supports push %ebp; movl %esp, %ebp [translation:ERROR] make: *** [getargs.gcmap] Error 1 [translation:ERROR] make: *** Waiting for unfinished jobs.... [translation:ERROR] """) [translation] start debugger... > /home/svenstaro/src/pypy/rpython/translator/platform/__init__.py(151)_handle_err o r() -> raise CompilationError(stdout, stderr) It works on i686, though I didn't test its function on i686, only that is translates. ---------- messages: 6879 nosy: pypy-issue, svenstaro priority: critical release: 2.3 status: unread title: pypy 2.3 doesn't translate with gcc 4.9 on x86_64 ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1764> ________________________________________

2 1

[issue1523] youtube-dl stuck at the end of HTTPS stream
by Grom PE 12 May '14

12 May '14

New submission from Grom PE <i(a)grompe.org.ru>: youtube-dl version 2013.05.05 up to newest 2013.06.34.4 when run with pypy 2.0.2 gets stuck at the end of HTTPS stream, which can be seen in the console as: $ pypy youtube-dl 7-tNUur2YoU [youtube] Setting language [youtube] 7-tNUur2YoU: Downloading video webpage Changing the line 413 in file "youtube_dl/extractor/youtube.py" to have http link instead makes the stream read correctly: url = 'http://www.youtube.com/watch?v=%s&gl=US&hl=en&has_verified=1' % video_id If left as https, it gets stuck at the line 416: video_webpage_bytes = compat_urllib_request.urlopen(request).read() While using Python 2.7 and 3.3 youtube-dl has no such problem. Other HTTPS streams seem to be read normally. Correct behavior: $ pypy youtube-dl 7-tNUur2YoU [youtube] Setting language [youtube] 7-tNUur2YoU: Downloading video webpage [youtube] 7-tNUur2YoU: Downloading video info webpage [youtube] 7-tNUur2YoU: Extracting video information [download] Destination: Goldfish - We Come Together (OFFICIAL)-7-tNUur2YoU.mp4 [download] 0.2% of 66.60MiB at 172.09KiB/s ETA 06:35 youtube-dl: http://rg3.github.io/youtube-dl/download.html ---------- messages: 5888 nosy: Grom PE, pypy-issue priority: bug release: 2.0 status: unread title: youtube-dl stuck at the end of HTTPS stream ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1523> ________________________________________

1 3

[issue1766] numpy.load fails because numpy.frombuffer is None
by Rich Wareham 12 May '14

12 May '14

New submission from Rich Wareham <rich.bugs.pypy(a)richwareham.com>: Using PyPy 2.3 installed via pyenv, np.load fails when trying to read data because it makes use of np.frombuffer which is None. Setting np.frombuffer equal to np.fromstring is a workaround. Numpy instaled via pip install git+https://bitbucket.org/pypy/numpy.git. Output of python --version: $ python --version Python 2.7.6 (394146e9bb67, May 08 2014, 17:08:54) [PyPy 2.3.0 with GCC 4.6.3] IPython log: # IPython log file get_ipython().magic(u'pinfo %save') get_ipython().magic(u'quickref') get_ipython().magic(u'pinfo %logstart') import numpy as np np.load('tests/lena.npz').keys() #[Out]# ['lena'] np.load('tests/lena.npz')['lena'] repr(np.frombuffer) #[Out]# 'None' np.frombuffer = np.fromstring np.load('tests/lena.npz')['lena'] #[Out]# array([[ 0.63529414, 0.63529414, 0.63529414, ..., 0.66666669, #[Out]# 0.60784316, 0.50196081], #[Out]# [ 0.63529414, 0.63529414, 0.63529414, ..., 0.66666669, #[Out]# 0.60784316, 0.50196081], #[Out]# [ 0.63529414, 0.63529414, 0.63529414, ..., 0.66666669, #[Out]# 0.60784316, 0.50196081], #[Out]# ..., #[Out]# [ 0.16862746, 0.16862746, 0.19607843, ..., 0.40784314, #[Out]# 0.39215687, 0.38431373], #[Out]# [ 0.17254902, 0.17254902, 0.21568628, ..., 0.40784314, #[Out]# 0.41176471, 0.42352942], #[Out]# [ 0.17254902, 0.17254902, 0.21568628, ..., 0.40784314, #[Out]# 0.41176471, 0.42352942]], dtype=float32) get_ipython().magic(u'logstop') Exception and traceback: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-5-95e5d6d28212> in <module>() ----> 1 np.load('tests/lena.npz')['lena'] /users/rjw57/projects/dtcwt/venv-pypy/site-packages/numpy/lib/npyio.pyc in __getitem__(self, key) 248 if magic == format.MAGIC_PREFIX: 249 bytes = self.zip.open(key) --> 250 return format.read_array(bytes) 251 else: 252 return self.zip.read(key) /users/rjw57/projects/dtcwt/venv-pypy/site-packages/numpy/lib/format.pyc in read_array(fp) 466 data = _read_bytes(fp, read_size, "array data") 467 array[i:i+read_count] = numpy.frombuffer(data, dtype=dtype, --> 468 count=read_count) 469 470 if fortran_order: TypeError: 'NoneType' object is not callable ---------- messages: 6882 nosy: pypy-issue, rjw57 priority: bug status: unread title: numpy.load fails because numpy.frombuffer is None ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1766> ________________________________________

1 0

[issue1765] __len__ can't return big numbers
by Ram Rachum 12 May '14

12 May '14

New submission from Ram Rachum <ram(a)rachum.com>: I have just opened the same issue on CPython, and it was closed as a limitation of the implementation: http://bugs.python.org/issue21444? @ok_message=msg%20217957%20created%0Aissue%2021444%20created&@template=item Maybe Pypy-dev would like to implement this. I want to use big numbers for an object's length, but currently Pypy doesn't support this: Python 2.7.6 (394146e9bb67, May 08 2014, 18:18:57) [PyPy 2.3.0 with MSC v.1500 32 bit] on win32 Type "help", "copyright", "credits" or "license" for more information. >>>> class A(object): .... def __len__(self): .... return 10 ** 100 .... >>>> a = A() >>>> len(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: long int too large to convert to int Same thing happened on Pypy3: Python 3.2.3 (d63636b30cc0, Jul 30 2013, 07:02:44) [PyPy 2.1.0-beta1 with MSC v.1500 32 bit] on win32 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``it's beautiful: very rectangular and yellow'' >>>> class A(object): .... def __len__(self): .... return 10 ** 100 .... >>>> a = A() >>>> len(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: long int too large to convert to int ---------- messages: 6880 nosy: cool-RR, pypy-issue priority: feature status: unread title: __len__ can't return big numbers ________________________________________ PyPy bug tracker <tracker(a)bugs.pypy.org> <https://bugs.pypy.org/issue1765> ________________________________________

2 1