[New-bugs-announce] [issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

New submission from Michael Osipov <1983-01-06 at gmx.net>:

Running from 3.7 branch on HP-UX 11.31 ia64, 32 bit, big endian.
The test output is:
> Re-running failed tests in verbose mode
> Re-running test 'test_utf8_mode' in verbose mode
> test_cmd_line (test.test_utf8_mode.UTF8ModeTests) ... FAIL
> test_env_var (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_filesystemencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_io (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_io_encoding (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_locale_getpreferredencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_optim_level (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_posix_locale (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_stdio (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_xoption (test.test_utf8_mode.UTF8ModeTests) ... ok
> 
> ======================================================================
> FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 230, in test_cmd_line
>     check('utf8=0', [c_arg], LC_ALL='C')
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check
>     self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
> - ['h\xc3\xa9\xe2\x82\xac']
> + ['h\udcc3\udca9\udce2\udc82\udcac']
>  : roman8:['h\xc3\xa9\xe2\x82\xac']
> 
> ----------------------------------------------------------------------
> Ran 10 tests in 2.595s
> 
> FAILED (failures=1)
> test test_utf8_mode failed
> 1 test failed again:
>     test_utf8_mode
> 
> == Tests result: FAILURE then FAILURE ==
> 
> 1 test failed:
>     test_utf8_mode
> 
> 1 re-run test:
>     test_utf8_mode
> 
> Total duration: 7 sec 265 ms
> Tests result: FAILURE then FAILURE
> Makefile:1066: recipe for target 'test' failed
> gmake: *** [test] Error 2
> 

I tried to understand the issue, but my Python knowledge is too low, especially I do not understand by a byte array "arg = 'h\xe9\u20ac'.encode('utf-8')" is passed as one arg to the forked process.

I highly assume that this is related to the non-standard, default character encoding on HP-UX: https://en.wikipedia.org/wiki/HP_Roman#HP_Roman-8 (roman8).

A stupid 8 bit encoding. The very same snippet on FreeBSD says:
> $ LC_ALL=C python3.6 test_utf8.py
> US-ASCII:[]

Willing to test and modify if someone tells what to do.

----------
components: Library (Lib), Tests
messages: 323516
nosy: michael-o
priority: normal
severity: normal
status: open
title: test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions
type: behavior
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34403>
_______________________________________