[New-bugs-announce] [issue42325] UnicodeDecodeError executing ./setup.py during build

[Skip Montanaro]

New submission from Skip Montanaro <skip.montanaro at gmail.com>:

I recently replaced Ubuntu 20.04 with Manjaro 20.2. In the process my Python builds broke in the sharedmods target of the Makefile. The tail end of the traceback is:

  File "/home/skip/src/python/cpython/./setup.py", line 246, in grep_headers_for
    if function in f.read():
  File "/home/skip/src/python/cpython/Lib/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 1600: invalid start byte

The grep_headers_for() function in setup.py appeared to be the culprit, so I added a print statement to its loop:

def grep_headers_for(function, headers):
    for header in headers:
        print("***", header, file=sys.stderr)
        with open(header, 'r') as f:
            if function in f.read():
                return True
    return False

which printed these lines:

*** /usr/include/umfpack_report_perm.h
*** /usr/include/dbstl_dbc.h
*** /usr/include/itclTclIntStubsFcn.h
*** /usr/include/dbstl_vector.h
*** /usr/include/cholmod_blas.h
*** /usr/include/amd.h
*** /usr/include/m17n-X.h

Sure enough, that m17n-X.h file (attached) doesn't contain utf-8 (my environment's encoding). According to the Emacs coding cookie at the end, the file is euc-japan encoded. Would simply catching the exception in grep_headers_for() be the correct way to deal with this?

----------
components: Build
files: m17n-X.h
messages: 380761
nosy: skip.montanaro
priority: normal
severity: normal
status: open
title: UnicodeDecodeError executing ./setup.py during build
versions: Python 3.10
Added file: https://bugs.python.org/file49593/m17n-X.h

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42325>
_______________________________________