Mailman 3 November 2016 - pypy-issue

Issue #2441: Unicode surrogate codepoints in string literals combined when using unittest module (pypy/pypy)
by byllyfish 30 Nov '16

30 Nov '16

New issue 2441: Unicode surrogate codepoints in string literals combined when using unittest module https://bitbucket.org/pypy/pypy/issues/2441/unicode-surrogate-codepoints-in… byllyfish: When running unit tests under PyPy3, I sometimes see unicode literals containing surrogate pairs combined into a single non-BMP character. Here is `test_surrogate.py`: ```python import unittest class TestSurrogate(unittest.TestCase): def test_surrogate(self): s = '\ud800\udc00' if len(s) != 2: raise ValueError(s.encode('raw-unicode-escape')) ``` The first time I run it, it works fine. ```bash $ ~/pypy3-v5.5.0-osx64/bin/pypy3 -m unittest test_surrogate . ---------------------------------------------------------------------- Ran 1 test in 0.000s OK ``` When I run the test a second time, it fails. The surrogate pair has been replaced. ```bash $ ~/pypy3-v5.5.0-osx64/bin/pypy3 -m unittest test_surrogate E ====================================================================== ERROR: test_surrogate (test_surrogate.TestSurrogate) ---------------------------------------------------------------------- Traceback (most recent call last): File "./test_surrogate.py", line 7, in test_surrogate raise ValueError(s.encode('raw-unicode-escape')) ValueError: b'\\U00010000' ---------------------------------------------------------------------- Ran 1 test in 0.010s FAILED (errors=1) ``` The failures continue until I touch the file. After that, the test will succeed the first time, then fail subsequently. If I touch the file and run pypy3 with -B (don't write .py[co] files on import), all the test runs succeed. N.B. The problem does NOT occur under normal conditions. I've only seen it using unittest. ``` # This small program always works fine! s = '\ud800\udc00' if len(s) != 2: raise ValueError(s.encode('raw-unicode-escape')) ``` I am running on Mac OS X 10.11.6. Please let me know if you can reproduce this. ``` $ ~/pypy3-v5.5.0-osx64/bin/pypy3 --version Python 3.3.5 (619c0d5af0e5, Oct 08 2016, 22:08:19) [PyPy 5.5.0-alpha0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] ```

1 0

Issue #2440: extension_suffixes() without cpyext (pypy/pypy)
by Richard Plangger 28 Nov '16

28 Nov '16

New issue 2440: extension_suffixes() without cpyext https://bitbucket.org/pypy/pypy/issues/2440/extension_suffixes-without-cpye… Richard Plangger: The condtion in the file `pypy/module/imp/interp_imp.py:~10` prevents pypy/tools/build_cffi_imports.py to succeed if pypy was compiled without cpyext module. >From my understanding it should be fine to just remove the condition and always include the result of `importing.get_so_extension(space)`.

1 0

Issue #2439: Pypy3.5 - pip install aiohttp fails (pypy/pypy)
by Maciej Fijalkowski 28 Nov '16

28 Nov '16

New issue 2439: Pypy3.5 - pip install aiohttp fails https://bitbucket.org/pypy/pypy/issues/2439/pypy35-pip-install-aiohttp-fails Maciej Fijalkowski: With: AttributeError: '_ssl._SSLSocket' object has no attribute 'server_hostname'

1 0

Issue #2438: Crash with pyglet on Windows (only with JIT) (pypy/pypy)
by Daniil Yarancev 27 Nov '16

27 Nov '16

New issue 2438: Crash with pyglet on Windows (only with JIT) https://bitbucket.org/pypy/pypy/issues/2438/crash-with-pyglet-on-windows-on… Daniil Yarancev: Trying to run https://github.com/fogleman/Minecraft on Windows with pypy, using pyglet. Without JIT (--jit off) everything works, but game is loading very slowly (you need to wait like 2-4 minutes have a possibility to move mouse, but further everything works OK). With JIT it start, you can move the mouse, BUT if you move on another block (if you will look at the ground, you will see block selection), PyPy crashes. Steps to reproduce: 1) Install pyglet ``` #!bash pypy -m pip install pyglet ``` 2) Download fogleman's Minecraft (written using pyglet) ``` #!bash git clone https://github.com/fogleman/Minecraft cd Minecraft ``` 3) Run it ``` #!bash pypy main.py ```

1 0

Issue #2437: cpyext: support PyModule_GetName (pypy/pypy)
by Wenzel Jakob 24 Nov '16

24 Nov '16

New issue 2437: cpyext: support PyModule_GetName https://bitbucket.org/pypy/pypy/issues/2437/cpyext-support-pymodule_getname Wenzel Jakob: The cpyext implementation of PyModule_GetName throws an unsupported operation exception. This one should be simple to do -- afaik it just returns ``__name__``.

1 0

Issue #2436: Support pybind11 in conjunction with PyPy's cpyext: Crash in PyDict_Next (pypy/pypy)
by Wenzel Jakob 24 Nov '16

24 Nov '16

New issue 2436: Support pybind11 in conjunction with PyPy's cpyext: Crash in PyDict_Next https://bitbucket.org/pypy/pypy/issues/2436/support-pybind11-in-conjunction… Wenzel Jakob: Hi, following the fix in Issue #2434, I was able to get basic types & methods to export properly in [pybind11](https://github.com/pybind/pybind11). Also, the entire test suite compiles now (it segfaults when actually running it, but still..). Yay! :) Looking into the test suite crashes, it turns out that the culprit is a call to ``PyDict_Next()`` to traverse the contents of a newly created type. Pybind11 uses this function to iterate over the dictionary of a type when binding C++ enumerations in Python (to do some basic postprocessing of the enumeration entries). The CPython API calls boil down to this: ```cpp PyObject *dict = ((PyTypeObject *) ... /* Pointer to newly created type */)->tp_dict; PyObject *key, *value; ssize_t pos = 0; while (PyDict_Next(dict, &pos, &key, &value)) { /* Do something -- Never gets here because PyDict_Next crashes */ } ``` This works fine in Python 2.7 and 3.x but crashes in PyPy. What's weird is that ``dict`` is a perfectly good dictionary even in PyPy. I can for instance turn it into a string and print it before the ``PyDict_Next()`` call, getting ```python {'__init__': <unbound method pybind11_tests.EMode.__init__>, '__new__': <builtin_function_or_method object at 0x0000000104754250>, '__dict__': <getset_descriptor object at 0x00000001019724d8>, '__weakref__': <getset_descriptor object at 0x000000010191b068>, '__doc__': None, '__module__': 'pybind11_tests', '__repr__': <unbound method pybind11_tests.EMode.__repr__>, '__int__': <unbound method pybind11_tests.EMode.__int__>, '__eq__': <unbound method pybind11_tests.EMode.__eq__>, '__ne__': <unbound method pybind11_tests.EMode.__ne__>, '__hash__': <unbound method pybind11_tests.EMode.__hash__>, '__getstate__': <unbound method pybind11_tests.EMode.__getstate__>, '__setstate__': <unbound method pybind11_tests.EMode.__setstate__>, 'EFirstMode': EMode.EFirstMode, 'ESecondMode': EMode.ESecondMode} ``` but ``PyDict_Next()`` still crashes. LLDB reports the following backtrace: ``` * frame #0: 0x0000000000000000 frame #1: 0x000000010062c0a4 libpypy-c.dylib`pypy_g_PyDict_Next + 244 frame #2: 0x00000001005f479b libpypy-c.dylib`pypy_g_wrapper_second_level__star_4_4 + 347 frame #3: 0x0000000105d8b0e0 pybind11_tests.pypy-41.so`pybind11::enum_<MyEnum>::export_values(this=0x00007fff5fbfe580) + 80 at pybind11.h:1260 ``` which is bizarre. If I read that correctly, PyPy seems to have done a jump to NULL. How to reproduce on your end: ``` $ git clone https://github.com/wjakob/pybind11 $ cd pybind11 $ cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-pypy> $ make $ pypy -m pip install pytest $ pypy -m pytest $ ``` Thank you, Wenzel

1 0

Issue #2435: Difference between PyPy and CPython when converting a subclass of dict to kwargs (pypy/pypy)
by Mark Heppner 23 Nov '16

23 Nov '16

New issue 2435: Difference between PyPy and CPython when converting a subclass of dict to kwargs https://bitbucket.org/pypy/pypy/issues/2435/difference-between-pypy-and-cpy… Mark Heppner: In Django, there is a data structure called a [`MultiValueDict`](https://github.com/django/django/blob/1.10.3/django/utils/datastructures.py#L48) that subclasses off of `dict`. When converting it to kwargs, it behaves differently than CPython. I submitted a [ticket to Django](https://code.djangoproject.com/ticket/27521) that was closed pretty quickly. An example using a [`QueryDict`](https://github.com/django/django/blob/1.10.3/django/http/request.py#L355): ```python from django.conf import settings settings.configure() from django.http.request import QueryDict q = QueryDict('foo=bar&foo=baz') def test(**kwargs): print(kwargs) test(**q) ``` In CPython, this returns `{u'foo': [u'bar', u'baz']}`, but in PyPy `{u'foo': u'baz'}` is returned. This only happens on conversion from the subclassed `dict` to kwargs. I'm not sure if this is a bug or if the docs already [explain this behavior](http://doc.pypy.org/en/latest/cpython_differences.html#mutating-c….

1 0

Issue #2434: Support pybind11 in conjunction with PyPy's cpyext: issue with heap types (pypy/pypy)
by Wenzel Jakob 23 Nov '16

23 Nov '16

New issue 2434: Support pybind11 in conjunction with PyPy's cpyext: issue with heap types https://bitbucket.org/pypy/pypy/issues/2434/support-pybind11-in-conjunction… Wenzel Jakob: Hi, I'm the maintainer of [pybind11](https://github.com/pybind/pybind11), which is a modern library for creating bindings to C++ code (similar in spirit to Boost.Python). Hearing of the cpyext improvements in the latest release, I tried to see if pybind11 would work with the supported CPython API in PyPy. This would be huge, since it immediately make many projects available in PyPy. The good news: I just had to make a few minor changes to get a few simple examples to compile. This is all in the WIP branch in pybind11 here: https://github.com/pybind/pybind11/pull/521 The bad news: I'm running into fundamental issues involving creation of types. To reproduce: ``` $ git clone https://github.com/wjakob/pybind11 $ cd pybind11 $ cat > test.cpp <<EOF #include <pybind11/pybind11.h> namespace py = pybind11; PYBIND11_PLUGIN(test) { py::module m("test"); struct Test { }; py::class_<Test>(m, "Test"); return m.ptr(); } EOF $ g++ test.cpp -g3 -std=c++11 -I <path-to-pypy-include> -I include -undefined dynamic_lookup -shared -rdynamic -o test.so ``` Loading the module from PyPy leads to the following error: ```bash $ pypy Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 13 2016, 01:57:41) [PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>> import cpyext >>>> pyext.load_module('test.so', 'test') Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: global name 'pyext' is not defined >>>> cpyext.load_module('test.so', 'test') Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: TypeError: can't set attributes on type object 'Test' ``` The error message is thrown when pybind11 basically calls (via CPython API) ```python setattr(Test, '__module__', 'test') ``` Digging into the PyPy source code reveals the following source of this error message: ```python def setdictvalue(self, space, name, w_value): if not self.is_heaptype(): raise oefmt(space.w_TypeError, "can't set attributes on type object '%N'", self) ``` So PyPy believes that ``Test`` is not a heap type. Which is confusing, because ``cpyext`` believes it is. The CPython API calls boil down to: ```cpp PyHeapTypeObject *type = ((PyHeapTypeObject*) PyType_Type.tp_alloc(&PyType_Type, 0) type->ht_type.tp_flags |= Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HEAPTYPE; .. fill out remaining type fields like name, docs, ... PyType_Ready(&type->ht_type); PyObject_SetAttrString(type->ht_type, "__module__", ...); // BOOM, this fails ``` Interestingly, the type instance returned by ``tp_alloc`` *already* has the heap type flag set, and it is still set after the ``PyType_Ready`` call. I can only assume that PyPy has its own heap type flag that somehow isn't synchronized with the types created by cpyext.

1 0

Issue #2433: Numpy segfault with PyPy 5.6.0 on OS X (pypy/pypy)
by Ralf Gommers 19 Nov '16

19 Nov '16

New issue 2433: Numpy segfault with PyPy 5.6.0 on OS X https://bitbucket.org/pypy/pypy/issues/2433/numpy-segfault-with-pypy-560-on… Ralf Gommers: This is with current numpy master on OS X 10.12.1. OS X binary from pypy.org: https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.6.0-osx64.tar.bz2 Installed numpy with ``./pypy-xxx/bin/pip install cython git+https://github.com/numpy/numpy.git``. $ ./pypy Python 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49) [PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy as np >>>> np.test() Running unit tests for numpy NumPy version 1.13.0.dev0+ec02bdc NumPy relaxed strides checking option: True NumPy is installed in /Users/rgommers/Downloads/pypy2-v5.6.0-osx64/site-packages/numpy Python version 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49)[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] nose version 1.3.7 .................................S.......................................................E........................................................................................S.............................................................RPython traceback: File "pypy_interpreter.c", line 43348, in BuiltinCodePassThroughArguments1_funcrun_obj File "pypy_module_cpyext_4.c", line 16627, in generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_17 Fatal RPython error: AssertionError Abort trap: 6 Compilers are the default Clang and Gfortran from Homebrew: $ clang --version Apple LLVM version 8.0.0 (clang-800.0.42.1) Target: x86_64-apple-darwin16.1.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin $ gfortran --version gfortran: warning: couldn't understand kern.osversion '16.1.0 GNU Fortran (Homebrew gcc 4.9.1) 4.9.1

1 0

Issue #2432: problems with pypy on osx, dyld: symbol not found _clock_gettime (pypy/pypy)
by Irmen de Jong 15 Nov '16

15 Nov '16

New issue 2432: problems with pypy on osx, dyld: symbol not found _clock_gettime https://bitbucket.org/pypy/pypy/issues/2432/problems-with-pypy-on-osx-dyld-… Irmen de Jong: I'm using homebrew on OSX 10.11.6 El capitan. Tonight I used it to upgrade to pypy 5.6.0 but this is now causing dynamic linking symbol issues. I think it is caused by the fact that they offer a compiled pypy version that was created using the version 8 compiler tools on OSX 10.12 Sierra and I'm running it on the previous OSX version. Some googling results in dozens of projects having dyld symbol issues related to this, for instance https://github.com/civetweb/civetweb/issues/349 I was wondering if this is something that homebrew should fix? Or is it something that should be fixed in pypy itself? The exact error I'm getting: ``` #!python phobos:~ irmen$ pypy Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 13 2016, 01:57:35) [PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>> import time >>>> time.clock() dyld: lazy symbol binding failed: Symbol not found: _clock_gettime Referenced from: /usr/local/Cellar/pypy/5.6.0/libexec/lib/libpypy-c.dylib Expected in: flat namespace dyld: Symbol not found: _clock_gettime Referenced from: /usr/local/Cellar/pypy/5.6.0/libexec/lib/libpypy-c.dylib Expected in: flat namespace Trace/BPT trap: 5 phobos:~ irmen$ ```

1 0