[pypy-issue] Issue #3165: PyPy should set `errors="surrogatepass"` when decoding numpy arrays to unicode scalars (pypy/pypy)

Eric Wieser issues-reply at bitbucket.org
Tue Feb 4 11:18:08 EST 2020


New issue 3165: PyPy should set `errors="surrogatepass"` when decoding numpy arrays to unicode scalars
https://bitbucket.org/pypy/pypy/issues/3165/pypy-should-set-errors-surrogatepass-when

Eric Wieser:

See [https://github.com/numpy/numpy/issues/15363](https://github.com/numpy/numpy/issues/15363), where the bug was reported in numpy.

The fix in numpy was to replace calls to `PyUnicode_DecodeUTF32` with calls to `PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, ...)`.

However, it seems that PyPI does not use our code there, and has their own implementation. Assuming `PyUnicode_FromKindAndData` is not available, the equivalent python code would be `some_bytes.decode('utf-32-le', errors='surrogatepass')`.




More information about the pypy-issue mailing list