Mailman 3 April 2018 - pypy-issue

Issue #2814: _pypyjson should use SIMD when available to skip whitespace characters faster (pypy/pypy)
by Omer Katz 29 Apr '18

29 Apr '18

New issue 2814: _pypyjson should use SIMD when available to skip whitespace characters faster https://bitbucket.org/pypy/pypy/issues/2814/_pypyjson-should-use-simd-when-… Omer Katz: RapidJSON has implemented whitespace skipping using a variety of SIMD interstices. For [SSE 4.2](https://github.com/Tencent/rapidjson/blob/master/include/rapidjson/rea…, [SSE 2](https://github.com/Tencent/rapidjson/blob/master/include/rapidjson/reade… and [NEON for ARM](https://github.com/Tencent/rapidjson/blob/master/include/rapidjson/rea…. We can port these functions to C pretty easily and the license of RapidJSON is MIT so that's allowed. The challenge is to detect if the user running PyPy has support for such instructions.

1 0

Issue #2813: seems bug in ctypes (pypy/pypy)
by 刘穆清 27 Apr '18

27 Apr '18

New issue 2813: seems bug in ctypes https://bitbucket.org/pypy/pypy/issues/2813/seems-bug-in-ctypes 刘穆清: ``` #!python from ctypes import * class Context(Structure): pass PtrTypeOfContext = POINTER(Context) LIB = cdll.LoadLibrary("./test.so") LIB.test.restype = Context LIB.test.argtypes = [] Context._fields_ = [ ("a", c_char_p), ("b", c_char_p), ("c", c_char_p), ("d", c_char_p), ] c = Context() v = LIB.test(c) print (v.a) print (v.b) print (v.c) print (v.d) ``` this code can be run in CPython, but in pypy, it will raise `SystemError: not supported by libffi` and If I comment line 5, pypy will be good

1 0

Issue #2812: json.loads fails on bytes (pypy/pypy)
by Stefane Fermigier 27 Apr '18

27 Apr '18

New issue 2812: json.loads fails on bytes https://bitbucket.org/pypy/pypy/issues/2812/jsonloads-fails-on-bytes Stefane Fermigier: ``` ➜ cat jsontest.py import json print(json.loads("1")) print(json.loads(b"1")) ➜ python jsontest.py 1 1 ➜ python3 jsontest.py 1 1 ➜ pypy jsontest.py 1 1 ➜ pypy3 jsontest.py 1 Traceback (most recent call last): File "jsontest.py", line 3, in <module> print(json.loads(b"1")) File "/usr/local/Cellar/pypy3/6.0.0/libexec/lib-python/3/json/__init__.py", line 318, in loads s.__class__.__name__)) TypeError: the JSON object must be str, not 'bytes' ```

1 0

Issue #2811: ropenssl.py incorrectly assumes presence of comp.h (pypy/pypy)
by Dominyk Tiller 26 Apr '18

26 Apr '18

New issue 2811: ropenssl.py incorrectly assumes presence of comp.h https://bitbucket.org/pypy/pypy/issues/2811/ropensslpy-incorrectly-assumes-… Dominyk Tiller: There's an erroneous assumption in `ropenssl.py` that the OpenSSL header `comp.h` will always be available, and consequently when that header isn't available the build can fail: ``` File "/private/tmp/pypy-20180426-89170-ltfanb/pypy2-v6.0.0-src/rpython/rtyper/tool/rffi_platform.py", line 844, in configure_external_library raise last_error [translation:ERROR] CompilationError: CompilationError(err=""" /private/tmp/pypy-20180426-89170-ltfanb/pypy2-v6.0.0-src/usession-release-pypy2.7-v6.0.0-0/platcheck_77.c:87:10: fatal error: 'openssl/comp.h' file not found #include <openssl/comp.h> ^~~~~~~~~~~~~~~~ 1 error generated. """) ``` Whilst it is *usually* available, it is perfectly legitimate to build openssl with `no-comp` passed during compile, which apparently disables installing that header entirely. Homebrew, the macOS package manager, has recently switched its OpenSSL formula to build with `no-comp`, which has revealed this issue. Normally in C you could wrap the header inclusion in something like `#ifndef OPENSSL_NO_COMP` but I'm not sure how to achieve that in Python.

1 0

Issue #2810: Embedding PyPy + cpyext API calls (pypy/pypy)
by Omer Ben-Amram 26 Apr '18

26 Apr '18

New issue 2810: Embedding PyPy + cpyext API calls https://bitbucket.org/pypy/pypy/issues/2810/embedding-pypy-cpyext-api-calls Omer Ben-Amram: Hi guys, I've been fiddling around with adding pypy support for the rust [PyO3](https://github.com/PyO3/pyo3). I've been quite successful with the original intent of generating PyPy compatible rust extensions! (wrapped as cpython modules) (the modifications are at - [my fork](https://github.com/omerbenamram/pyo3/tree/feature/add-build-pypy-supp…) by correctly linking to libpypy, changing the exported names and re implementing some missing macros in rust. However, the other direction (rust code that embeds pypy) - which would try to call cpyext functions directly from rust works for cpython (since PyO3 handles all cpython initialization correctly), but crashes with PyPy. I'm linking my rust library against a debug version of libpypy (5.10, haven't downloaded 6 yet) I've tried several combinations of calls to `rpython_startup_code`, `pypy_setup_home` (as referenced [here](http://doc.pypy.org/en/latest/embedding.html)), all of them yielded the following results: So something like this works (will exit without crashing): ![Screen Shot 2018-04-26 at 13.57.59.png](https://bitbucket.org/repo/R7AbB/images/1079889783-Screen%20S… But importing via regular API would crash, interestingly after printing the zen: ![Screen Shot 2018-04-26 at 14.00.51.png](https://bitbucket.org/repo/R7AbB/images/3631796291-Screen%20S… ![Screen Shot 2018-04-26 at 14.02.22.png](https://bitbucket.org/repo/R7AbB/images/626902820-Screen%20Sh… Interestingly, using an API that doesn't allocate anything, such as `PyBool_FromLong` results in a more verbose crash. ![Screen Shot 2018-04-26 at 14.04.20.png](https://bitbucket.org/repo/R7AbB/images/2586941973-Screen%20S… I'm suspicious that there is some ref counting/GIL management issue that is causing the crashes. Am I missing something here? would it be possible to use the embedding API to call cpyext emulated functions from rust? A minimal code sample to reproduce the situation can be found [here](https://github.com/omerbenamram/pypy_ffi_playground/tree/pypy-embeddi… Thanks! Omer

1 0

Issue #2809: Unable to install libraries with c modules with custom rpath on certain OS (e.g. Arch) (pypy/pypy)
by Yusuke Tsutsumi 25 Apr '18

25 Apr '18

New issue 2809: Unable to install libraries with c modules with custom rpath on certain OS (e.g. Arch) https://bitbucket.org/pypy/pypy/issues/2809/unable-to-install-libraries-wit… Yusuke Tsutsumi: # Steps To Reproduce # 1. create a virtualenv for pypy (bin/virtualenv-pypy) 2. run: ``` $ pip install pyzmq --global-option="build_ext" --global-option="--rpath=./Lib" ``` The output will probably look like: ``` #!python creating build/lib.linux-x86_64-3.5/zmq/backend/cffi cc -pthread -shared -Wunused-parameter -L/home/tsutsumi/Downloads/pypy3/env/lib -Wl,-rpath,/home/tsutsumi/Downloads/pypy3/env/lib build/temp.linux-x86_64-3.5/zmq/backend/cffi/__pycache__/_cffi_ext.o -R./Lib -lzmq -o build/lib.linux-x86_64-3.5/zmq/backend/cffi/_cffi_ext.pypy3-510-x86_64-linux-gnu.so cc: error: unrecognized command line option ‘-R’ error: command 'cc' failed with exit status 1 ---------------------------------------- Command "/home/tsutsumi/Downloads/pypy3/env/bin/pypy -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-ncfdiwxd/pyzmq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" build_ext --rpath=./Lib install --record /tmp/pip-record-ky03ulyu/install-record.txt --single-version-externally-managed --compile --install-headers /home/tsutsumi/Downloads/pypy3/env/include/site/python3.5/pyzmq" failed with error code 1 in /tmp/pip-install-ncfdiwxd/pyzmq/ ``` # Investigation / Details # The --rpath parameter passed into the setup.py specifies additional parameters to be passed into GCC. Those parameters are chosen by the unixcompiler.py module (which seems to be the same for both pypy and cpython): https://github.com/python/cpython/blob/master/Lib/distutils/unixccompiler.py Which is choosing the erroneous flags -R in lieu of -Wl for gcc. It looks like this isn't encountered during cpython, as a result of gcc being noted as the compiler, and that forces the correct values: https://github.com/python/cpython/blob/master/Lib/distutils/unixccompiler.p… Running the sysconfig.get_config_var for CC gives nothing on pypy3: ``` $ pypy3 -c "import sysconfig; print(sysconfig.get_config_var('CC'))" None ``` Digging a bit deeper, it looks like these variables are autopopulated by consuming the Makefile used to compile Python. In pypy3, this file does not exist. The function responsible for populating the CC config var: https://github.com/python/cpython/blob/master/Lib/sysconfig.py#L411 The function reading the makefile, and creating the subsequent file that is consumed to construct the config var needed to resolve to the right cc variable: https://github.com/python/cpython/blob/master/Lib/sysconfig.py#L348 It seems like the solution could be providing a faux-makefile, or potentially patching or creating the final build file from which the variables are consumed. I'm happy to do some work in the area and send a PR, just not sure what the best direction would be to resolve this.

1 0

Issue #2808: crypt() not defined in unistd.h breaks build on Fedora 28 (pypy/pypy)
by pv 25 Apr '18

25 Apr '18

New issue 2808: crypt() not defined in unistd.h breaks build on Fedora 28 https://bitbucket.org/pypy/pypy/issues/2808/crypt-not-defined-in-unistdh-br… pv: Apparently Fedora (and probably also other systems) are using libxcrypt instead of glibc crypt(). This breaks pypy's crypt module build (translation fails with c compilation error), since crypt() is defined in crypt.h and no longer in unistd.h. https://fedoraproject.org/wiki/Changes/Replace_glibc_libcrypt_with_libxcrypt Not sure how pypy handles system-specific includes, probably just adding `includes=["crypt.h"]` is not the right thing on all systems.

1 0

Issue #2807: array.array fails to import in Cython (pypy/pypy)
by Roman Yurchak 22 Apr '18

22 Apr '18

New issue 2807: array.array fails to import in Cython https://bitbucket.org/pypy/pypy/issues/2807/arrayarray-fails-to-import-in-c… Roman Yurchak: The following Cython example, **example_array.pyx** ```py import array from cpython cimport array ``` that can be build with, ``` cythonize -a -i example_array.pyx ``` fails to import with PyPy 3.5.3 - 5.10.1 with the following traceback, ``` python -c 'import example_array' Traceback (most recent call last): File "<string>", line 1, in <module> File "array.pxd", line 58, in init example_array ValueError: array.array has the wrong size, try recompiling. Expected 24, got 72 ``` this was reported in the Cython project in https://github.com/cython/cython/issues/1818 It is also an issue that came up when building scikit-learn with PyPy https://github.com/scikit-learn/scikit-learn/pull/11010 cc @rlamy

1 0

Issue #2806: Calling tp_init does not call __init__ (pypy/pypy)
by Christoph Reiter 22 Apr '18

22 Apr '18

New issue 2806: Calling tp_init does not call __init__ https://bitbucket.org/pypy/pypy/issues/2806/calling-tp_init-does-not-call-_… Christoph Reiter: Minimal example extension: https://gist.github.com/lazka/871913fb1442d214e9f4d77d3ff7396c >From what I see doing a `PyObject_CallMethod (obj, "__init__", NULL);` works just as well, so I'll use that instead.

1 0

Issue #2805: AsyncActions can only .fire() once (pypy/pypy)
by Ronan Lamy 21 Apr '18

21 Apr '18

New issue 2805: AsyncActions can only .fire() once https://bitbucket.org/pypy/pypy/issues/2805/asyncactions-can-only-fire-once Ronan Lamy: After a88536ae8554, an AsyncAction cannot be fired again until the fired_actions queue (`AbstractActionFlag._fired_actions_first`/`AbstractActionFlag._fired_actions_last`) is cleared. However, when running `pytest pandas/tests/frame/test_to_csv.py -k chunking` with a pypy3 nightly, it appears that PyObjDeallocAction does exactly that, causing the assert in AbstractActionFlag._fired_actions_append() to fail.

1 0