Mailman 3 March 2021 - Python-checkins

bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075) (GH-25083)
by pablogsal 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/ff4715a7332123713de9920e724bb6c412… commit: ff4715a7332123713de9920e724bb6c412d8ccef branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-03-30T00:24:33+01:00 summary: bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075) (GH-25083) (cherry picked from commit 09b90a037d18f5d4acdf1b14082e57bda78e85d3) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst M Lib/test/test_sys.py M Python/pythonrun.c diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index fc6c1ef1acd7c..fce6b98299f78 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1417,6 +1417,21 @@ def test_asyncgen_hooks(self): self.assertIsNone(cur.firstiter) self.assertIsNone(cur.finalizer) + def test_changing_sys_stderr_and_removing_reference(self): + # If the default displayhook doesn't take a strong reference + # to sys.stderr the following code can crash. See bpo-43660 + # for more details. + code = textwrap.dedent(''' + import sys + class MyStderr: + def write(self, s): + sys.stderr = None + sys.stderr = MyStderr() + 1/0 + ''') + rc, out, err = assert_python_failure('-c', code) + self.assertEqual(out, b"") + self.assertEqual(err, b"") if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst new file mode 100644 index 0000000000000..98419501d9e7b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst @@ -0,0 +1,3 @@ +Fix crash that happens when replacing ``sys.stderr`` with a callable that +can remove the object while an exception is being printed. Patch by Pablo +Galindo. diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 04540989f3043..34cfb7f4b4fd7 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1072,8 +1072,9 @@ PyErr_Display(PyObject *exception, PyObject *value, PyObject *tb) if (file == Py_None) { return; } - + Py_INCREF(file); _PyErr_Display(file, exception, value, tb); + Py_DECREF(file); } PyObject *

1 0

[3.7] bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075). (GH-25085)
by pablogsal 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/2f01c562be913004e1d46fc02705c51b81… commit: 2f01c562be913004e1d46fc02705c51b81638d67 branch: 3.7 author: Pablo Galindo <Pablogsal(a)gmail.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-03-30T00:24:17+01:00 summary: [3.7] bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075). (GH-25085) (cherry picked from commit 09b90a037d18f5d4acdf1b14082e57bda78e85d3) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst M Lib/test/test_sys.py M Python/pythonrun.c diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 84927a393f175..0478f20cd33b4 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1292,6 +1292,21 @@ def test_asyncgen_hooks(self): self.assertIsNone(cur.firstiter) self.assertIsNone(cur.finalizer) + def test_changing_sys_stderr_and_removing_reference(self): + # If the default displayhook doesn't take a strong reference + # to sys.stderr the following code can crash. See bpo-43660 + # for more details. + code = textwrap.dedent(''' + import sys + class MyStderr: + def write(self, s): + sys.stderr = None + sys.stderr = MyStderr() + 1/0 + ''') + rc, out, err = assert_python_failure('-c', code) + self.assertEqual(out, b"") + self.assertEqual(err, b"") def test_main(): test.support.run_unittest(SysModuleTest, SizeofTest) diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst new file mode 100644 index 0000000000000..98419501d9e7b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst @@ -0,0 +1,3 @@ +Fix crash that happens when replacing ``sys.stderr`` with a callable that +can remove the object while an exception is being printed. Patch by Pablo +Galindo. diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 4c974cef39c78..b7b77d1f9ee3f 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -932,7 +932,9 @@ PyErr_Display(PyObject *exception, PyObject *value, PyObject *tb) seen = PySet_New(NULL); if (seen == NULL) PyErr_Clear(); + Py_INCREF(f); print_exception_recursive(f, value, seen); + Py_DECREF(f); Py_XDECREF(seen); } }

1 0

bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075)
by miss-islington 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/ba7f8638f93b5d999b25d8556ca19bdc2e… commit: ba7f8638f93b5d999b25d8556ca19bdc2e12f359 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2021-03-29T16:03:16-07:00 summary: bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075) (cherry picked from commit 09b90a037d18f5d4acdf1b14082e57bda78e85d3) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst M Lib/test/test_sys.py M Python/pythonrun.c diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index af0e54bd0e238..140c65aa3bfa2 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1441,6 +1441,21 @@ def test_asyncgen_hooks(self): self.assertIsNone(cur.firstiter) self.assertIsNone(cur.finalizer) + def test_changing_sys_stderr_and_removing_reference(self): + # If the default displayhook doesn't take a strong reference + # to sys.stderr the following code can crash. See bpo-43660 + # for more details. + code = textwrap.dedent(''' + import sys + class MyStderr: + def write(self, s): + sys.stderr = None + sys.stderr = MyStderr() + 1/0 + ''') + rc, out, err = assert_python_failure('-c', code) + self.assertEqual(out, b"") + self.assertEqual(err, b"") if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst new file mode 100644 index 0000000000000..98419501d9e7b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst @@ -0,0 +1,3 @@ +Fix crash that happens when replacing ``sys.stderr`` with a callable that +can remove the object while an exception is being printed. Patch by Pablo +Galindo. diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 3831e81ee9b89..c55e7674a665c 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1034,8 +1034,9 @@ PyErr_Display(PyObject *exception, PyObject *value, PyObject *tb) if (file == Py_None) { return; } - + Py_INCREF(file); _PyErr_Display(file, exception, value, tb); + Py_DECREF(file); } PyObject *

1 0

bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075)
by pablogsal 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/09b90a037d18f5d4acdf1b14082e57bda7… commit: 09b90a037d18f5d4acdf1b14082e57bda78e85d3 branch: master author: Pablo Galindo <Pablogsal(a)gmail.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-03-29T23:38:51+01:00 summary: bpo-43660: Fix crash when displaying exceptions with custom values for sys.stderr (GH-25075) files: A Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst M Lib/test/test_sys.py M Python/pythonrun.c diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 5b004c2b52da8..ee39375af3160 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1512,6 +1512,21 @@ def test_asyncgen_hooks(self): self.assertIsNone(cur.firstiter) self.assertIsNone(cur.finalizer) + def test_changing_sys_stderr_and_removing_reference(self): + # If the default displayhook doesn't take a strong reference + # to sys.stderr the following code can crash. See bpo-43660 + # for more details. + code = textwrap.dedent(''' + import sys + class MyStderr: + def write(self, s): + sys.stderr = None + sys.stderr = MyStderr() + 1/0 + ''') + rc, out, err = assert_python_failure('-c', code) + self.assertEqual(out, b"") + self.assertEqual(err, b"") if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst new file mode 100644 index 0000000000000..98419501d9e7b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-03-29-19-50-34.bpo-43660.scTgag.rst @@ -0,0 +1,3 @@ +Fix crash that happens when replacing ``sys.stderr`` with a callable that +can remove the object while an exception is being printed. Patch by Pablo +Galindo. diff --git a/Python/pythonrun.c b/Python/pythonrun.c index b562875f8a4fb..1715cde49dd68 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1082,8 +1082,9 @@ PyErr_Display(PyObject *exception, PyObject *value, PyObject *tb) if (file == Py_None) { return; } - + Py_INCREF(file); _PyErr_Display(file, exception, value, tb); + Py_DECREF(file); } PyObject *

1 0

bpo-43562: fix test_ssl to skip on unreachable network (GH-24937) (GH-25047)
by vstinner 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/20b2f2bfc2e72c6d9ed983d1f0aa7a141e… commit: 20b2f2bfc2e72c6d9ed983d1f0aa7a141e9b3fc3 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: vstinner <vstinner(a)python.org> date: 2021-03-29T21:37:38+02:00 summary: bpo-43562: fix test_ssl to skip on unreachable network (GH-24937) (GH-25047) This test checks result code of the connection directly, so it never raises an exception that can be suppressed by `support.transient_internet`. Directly support skipping the test in case of unreachable network. (cherry picked from commit 29c451c6989c3c94fa0a9facf187c24f3cbf2420) Co-authored-by: Carl Meyer <carl(a)oddbird.net> Co-authored-by: Carl Meyer <carl(a)oddbird.net> files: M Lib/test/test_ssl.py diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 3d995db6d1ea1..ec7bb74d28ce6 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2271,6 +2271,8 @@ def test_timeout_connect_ex(self): rc = s.connect_ex((REMOTE_HOST, 443)) if rc == 0: self.skipTest("REMOTE_HOST responded too quickly") + elif rc == errno.ENETUNREACH: + self.skipTest("Network unreachable.") self.assertIn(rc, (errno.EAGAIN, errno.EWOULDBLOCK)) @unittest.skipUnless(support.IPV6_ENABLED, 'Needs IPv6')

1 0

bpo-43562: fix test_ssl to skip on unreachable network (GH-24937) (GH-25046)
by vstinner 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/acb584958e215aff8fc8f5e2eb3fef481b… commit: acb584958e215aff8fc8f5e2eb3fef481b662f69 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: vstinner <vstinner(a)python.org> date: 2021-03-29T21:37:32+02:00 summary: bpo-43562: fix test_ssl to skip on unreachable network (GH-24937) (GH-25046) This test checks result code of the connection directly, so it never raises an exception that can be suppressed by `support.transient_internet`. Directly support skipping the test in case of unreachable network. (cherry picked from commit 29c451c6989c3c94fa0a9facf187c24f3cbf2420) Co-authored-by: Carl Meyer <carl(a)oddbird.net> Co-authored-by: Carl Meyer <carl(a)oddbird.net> files: M Lib/test/test_ssl.py diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 738929e3195bc..ae9a0b2efc985 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2276,6 +2276,8 @@ def test_timeout_connect_ex(self): rc = s.connect_ex((REMOTE_HOST, 443)) if rc == 0: self.skipTest("REMOTE_HOST responded too quickly") + elif rc == errno.ENETUNREACH: + self.skipTest("Network unreachable.") self.assertIn(rc, (errno.EAGAIN, errno.EWOULDBLOCK)) @unittest.skipUnless(socket_helper.IPV6_ENABLED, 'Needs IPv6')

1 0

bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843) (GH-24906)
by vstinner 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/3b6e61ee0812359029cac176042d9c835c… commit: 3b6e61ee0812359029cac176042d9c835c60f185 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: vstinner <vstinner(a)python.org> date: 2021-03-29T21:36:47+02:00 summary: bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843) (GH-24906) Python no longer fails at startup with a fatal error if a command line argument contains an invalid Unicode character. The Py_DecodeLocale() function now escapes byte sequences which would be decoded as Unicode characters outside the [U+0000; U+10ffff] range. Use MAX_UNICODE constant in unicodeobject.c. (cherry picked from commit 9976834f807ea63ca51bc4f89be457d734148682) Co-authored-by: Victor Stinner <vstinner(a)python.org> Co-authored-by: Victor Stinner <vstinner(a)python.org> files: A Misc/NEWS.d/next/Core and Builtins/2021-03-13-13-57-21.bpo-35883.UyGpdG.rst M Lib/test/test_cmd_line.py M Objects/unicodeobject.c M Python/fileutils.c diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py index a5ece9bd611f4..871a9c7a25529 100644 --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -190,38 +190,72 @@ def test_undecodable_code(self): if not stdout.startswith(pattern): raise AssertionError("%a doesn't start with %a" % (stdout, pattern)) + @unittest.skipIf(sys.platform == 'win32', + 'Windows has a native unicode API') + def test_invalid_utf8_arg(self): + # bpo-35883: Py_DecodeLocale() must escape b'\xfd\xbf\xbf\xbb\xba\xba' + # byte sequence with surrogateescape rather than decoding it as the + # U+7fffbeba character which is outside the [U+0000; U+10ffff] range of + # Python Unicode characters. + # + # Test with default config, in the C locale, in the Python UTF-8 Mode. + code = 'import sys, os; s=os.fsencode(sys.argv[1]); print(ascii(s))' + base_cmd = [sys.executable, '-c', code] + + def run_default(arg): + cmd = [sys.executable, '-c', code, arg] + return subprocess.run(cmd, stdout=subprocess.PIPE, text=True) + + def run_c_locale(arg): + cmd = [sys.executable, '-c', code, arg] + env = dict(os.environ) + env['LC_ALL'] = 'C' + return subprocess.run(cmd, stdout=subprocess.PIPE, + text=True, env=env) + + def run_utf8_mode(arg): + cmd = [sys.executable, '-X', 'utf8', '-c', code, arg] + return subprocess.run(cmd, stdout=subprocess.PIPE, text=True) + + valid_utf8 = 'e:\xe9, euro:\u20ac, non-bmp:\U0010ffff'.encode('utf-8') + # invalid UTF-8 byte sequences with a valid UTF-8 sequence + # in the middle. + invalid_utf8 = ( + b'\xff' # invalid byte + b'\xc3\xff' # invalid byte sequence + b'\xc3\xa9' # valid utf-8: U+00E9 character + b'\xed\xa0\x80' # lone surrogate character (invalid) + b'\xfd\xbf\xbf\xbb\xba\xba' # character outside [U+0000; U+10ffff] + ) + test_args = [valid_utf8, invalid_utf8] + + for run_cmd in (run_default, run_c_locale, run_utf8_mode): + with self.subTest(run_cmd=run_cmd): + for arg in test_args: + proc = run_cmd(arg) + self.assertEqual(proc.stdout.rstrip(), ascii(arg)) + @unittest.skipUnless((sys.platform == 'darwin' or support.is_android), 'test specific to Mac OS X and Android') def test_osx_android_utf8(self): - def check_output(text): - decoded = text.decode('utf-8', 'surrogateescape') - expected = ascii(decoded).encode('ascii') + b'\n' + text = 'e:\xe9, euro:\u20ac, non-bmp:\U0010ffff'.encode('utf-8') + code = "import sys; print(ascii(sys.argv[1]))" - env = os.environ.copy() - # C locale gives ASCII locale encoding, but Python uses UTF-8 - # to parse the command line arguments on Mac OS X and Android. - env['LC_ALL'] = 'C' + decoded = text.decode('utf-8', 'surrogateescape') + expected = ascii(decoded).encode('ascii') + b'\n' - p = subprocess.Popen( - (sys.executable, "-c", "import sys; print(ascii(sys.argv[1]))", text), - stdout=subprocess.PIPE, - env=env) - stdout, stderr = p.communicate() - self.assertEqual(stdout, expected) - self.assertEqual(p.returncode, 0) + env = os.environ.copy() + # C locale gives ASCII locale encoding, but Python uses UTF-8 + # to parse the command line arguments on Mac OS X and Android. + env['LC_ALL'] = 'C' - # test valid utf-8 - text = 'e:\xe9, euro:\u20ac, non-bmp:\U0010ffff'.encode('utf-8') - check_output(text) - - # test invalid utf-8 - text = ( - b'\xff' # invalid byte - b'\xc3\xa9' # valid utf-8 character - b'\xc3\xff' # invalid byte sequence - b'\xed\xa0\x80' # lone surrogate character (invalid) - ) - check_output(text) + p = subprocess.Popen( + (sys.executable, "-c", code, text), + stdout=subprocess.PIPE, + env=env) + stdout, stderr = p.communicate() + self.assertEqual(stdout, expected) + self.assertEqual(p.returncode, 0) def test_unbuffered_output(self): # Test expected operation of the '-u' switch diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-03-13-13-57-21.bpo-35883.UyGpdG.rst b/Misc/NEWS.d/next/Core and Builtins/2021-03-13-13-57-21.bpo-35883.UyGpdG.rst new file mode 100644 index 0000000000000..46742429db64b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-03-13-13-57-21.bpo-35883.UyGpdG.rst @@ -0,0 +1,4 @@ +Python no longer fails at startup with a fatal error if a command line +argument contains an invalid Unicode character. The +:c:func:`Py_DecodeLocale` function now escapes byte sequences which would be +decoded as Unicode characters outside the [U+0000; U+10ffff] range. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index a038e0d4c06e7..28f9c9bb0634b 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -90,7 +90,8 @@ NOTE: In the interpreter's initialization phase, some globals are currently extern "C" { #endif -/* Maximum code point of Unicode 6.0: 0x10ffff (1,114,111) */ +// Maximum code point of Unicode 6.0: 0x10ffff (1,114,111). +// The value must be the same in fileutils.c. #define MAX_UNICODE 0x10ffff #ifdef Py_DEBUG @@ -1707,8 +1708,8 @@ find_maxchar_surrogates(const wchar_t *begin, const wchar_t *end, *maxchar = ch; if (*maxchar > MAX_UNICODE) { PyErr_Format(PyExc_ValueError, - "character U+%x is not in range [U+0000; U+10ffff]", - ch); + "character U+%x is not in range [U+0000; U+%x]", + ch, MAX_UNICODE); return -1; } } @@ -13610,7 +13611,7 @@ _PyUnicodeWriter_PrepareKindInternal(_PyUnicodeWriter *writer, { case PyUnicode_1BYTE_KIND: maxchar = 0xff; break; case PyUnicode_2BYTE_KIND: maxchar = 0xffff; break; - case PyUnicode_4BYTE_KIND: maxchar = 0x10ffff; break; + case PyUnicode_4BYTE_KIND: maxchar = MAX_UNICODE; break; default: Py_UNREACHABLE(); } diff --git a/Python/fileutils.c b/Python/fileutils.c index fd2d5faa0fdf7..3e1311c978686 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -33,6 +33,13 @@ extern int winerror_to_errno(int); int _Py_open_cloexec_works = -1; #endif +// The value must be the same in unicodeobject.c. +#define MAX_UNICODE 0x10ffff + +// mbstowcs() and mbrtowc() errors +static const size_t DECODE_ERROR = ((size_t)-1); +static const size_t INCOMPLETE_CHARACTER = (size_t)-2; + static int get_surrogateescape(_Py_error_handler errors, int *surrogateescape) @@ -85,6 +92,57 @@ _Py_device_encoding(int fd) Py_RETURN_NONE; } + +static size_t +is_valid_wide_char(wchar_t ch) +{ + if (Py_UNICODE_IS_SURROGATE(ch)) { + // Reject lone surrogate characters + return 0; + } + if (ch > MAX_UNICODE) { + // bpo-35883: Reject characters outside [U+0000; U+10ffff] range. + // The glibc mbstowcs() UTF-8 decoder does not respect the RFC 3629, + // it creates characters outside the [U+0000; U+10ffff] range: + // https://sourceware.org/bugzilla/show_bug.cgi?id=2373 + return 0; + } + return 1; +} + + +static size_t +_Py_mbstowcs(wchar_t *dest, const char *src, size_t n) +{ + size_t count = mbstowcs(dest, src, n); + if (dest != NULL && count != DECODE_ERROR) { + for (size_t i=0; i < count; i++) { + wchar_t ch = dest[i]; + if (!is_valid_wide_char(ch)) { + return DECODE_ERROR; + } + } + } + return count; +} + + +#ifdef HAVE_MBRTOWC +static size_t +_Py_mbrtowc(wchar_t *pwc, const char *str, size_t len, mbstate_t *pmbs) +{ + assert(pwc != NULL); + size_t count = mbrtowc(pwc, str, len, pmbs); + if (count != 0 && count != DECODE_ERROR && count != INCOMPLETE_CHARACTER) { + if (!is_valid_wide_char(*pwc)) { + return DECODE_ERROR; + } + } + return count; +} +#endif + + #if !defined(_Py_FORCE_UTF8_FS_ENCODING) && !defined(MS_WINDOWS) #define USE_FORCE_ASCII @@ -151,8 +209,8 @@ check_force_ascii(void) size_t res; ch = (unsigned char)0xA7; - res = mbstowcs(&wch, (char*)&ch, 1); - if (res != (size_t)-1 && wch == L'\xA7') { + res = _Py_mbstowcs(&wch, (char*)&ch, 1); + if (res != DECODE_ERROR && wch == L'\xA7') { /* On HP-UX withe C locale or the POSIX locale, nl_langinfo(CODESET) announces "roman8", whereas mbstowcs() uses Latin1 encoding in practice. Force ASCII in this case. @@ -199,8 +257,8 @@ check_force_ascii(void) unsigned uch = (unsigned char)i; ch[0] = (char)uch; - res = mbstowcs(wch, ch, 1); - if (res != (size_t)-1) { + res = _Py_mbstowcs(wch, ch, 1); + if (res != DECODE_ERROR) { /* decoding a non-ASCII character from the locale encoding succeed: the locale encoding is not ASCII, force ASCII */ return 1; @@ -390,9 +448,9 @@ decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen, */ argsize = strlen(arg); #else - argsize = mbstowcs(NULL, arg, 0); + argsize = _Py_mbstowcs(NULL, arg, 0); #endif - if (argsize != (size_t)-1) { + if (argsize != DECODE_ERROR) { if (argsize > PY_SSIZE_T_MAX / sizeof(wchar_t) - 1) { return -1; } @@ -401,21 +459,13 @@ decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen, return -1; } - count = mbstowcs(res, arg, argsize + 1); - if (count != (size_t)-1) { - wchar_t *tmp; - /* Only use the result if it contains no - surrogate characters. */ - for (tmp = res; *tmp != 0 && - !Py_UNICODE_IS_SURROGATE(*tmp); tmp++) - ; - if (*tmp == 0) { - if (wlen != NULL) { - *wlen = count; - } - *wstr = res; - return 0; + count = _Py_mbstowcs(res, arg, argsize + 1); + if (count != DECODE_ERROR) { + *wstr = res; + if (wlen != NULL) { + *wlen = count; } + return 0; } PyMem_RawFree(res); } @@ -439,13 +489,13 @@ decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen, out = res; memset(&mbs, 0, sizeof mbs); while (argsize) { - size_t converted = mbrtowc(out, (char*)in, argsize, &mbs); + size_t converted = _Py_mbrtowc(out, (char*)in, argsize, &mbs); if (converted == 0) { /* Reached end of string; null char stored. */ break; } - if (converted == (size_t)-2) { + if (converted == INCOMPLETE_CHARACTER) { /* Incomplete character. This should never happen, since we provide everything that we have - unless there is a bug in the C library, or I @@ -453,32 +503,22 @@ decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen, goto decode_error; } - if (converted == (size_t)-1) { + if (converted == DECODE_ERROR) { if (!surrogateescape) { goto decode_error; } - /* Conversion error. Escape as UTF-8b, and start over - in the initial shift state. */ + /* Decoding error. Escape as UTF-8b, and start over in the initial + shift state. */ *out++ = 0xdc00 + *in++; argsize--; memset(&mbs, 0, sizeof mbs); continue; } - if (Py_UNICODE_IS_SURROGATE(*out)) { - if (!surrogateescape) { - goto decode_error; - } + // _Py_mbrtowc() reject lone surrogate characters + assert(!Py_UNICODE_IS_SURROGATE(*out)); - /* Surrogate character. Escape the original - byte sequence with surrogateescape. */ - argsize -= converted; - while (converted--) { - *out++ = 0xdc00 + *in++; - } - continue; - } /* successfully converted some bytes */ in += converted; argsize -= converted; @@ -655,7 +695,7 @@ encode_current_locale(const wchar_t *text, char **str, else { converted = wcstombs(NULL, buf, 0); } - if (converted == (size_t)-1) { + if (converted == DECODE_ERROR) { goto encode_error; } if (bytes != NULL) { @@ -1371,7 +1411,7 @@ _Py_wfopen(const wchar_t *path, const wchar_t *mode) char cmode[10]; size_t r; r = wcstombs(cmode, mode, 10); - if (r == (size_t)-1 || r >= 10) { + if (r == DECODE_ERROR || r >= 10) { errno = EINVAL; return NULL; }

1 0

bpo-43659: Fix test_curses on AIX (GH-25074)
by miss-islington 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/f1d53bcd536036c0a0c39962fbc0f465b5… commit: f1d53bcd536036c0a0c39962fbc0f465b5865492 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2021-03-29T12:31:01-07:00 summary: bpo-43659: Fix test_curses on AIX (GH-25074) curses.update_lines_cols() is only defined when the curses library provides either resizeterm() or resize_term() functions which are optional and are not provided on AIX. (cherry picked from commit c8b5738810516df5722caf049003e9b319427bec) Co-authored-by: Michael Felt <aixtools(a)users.noreply.github.com> files: M Lib/test/test_curses.py diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index c1ad23b456e20..4bdc2379fd1ed 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -1167,6 +1167,7 @@ def test_issue13051(self): class MiscTests(unittest.TestCase): + @requires_curses_func('update_lines_cols') def test_update_lines_cols(self): curses.update_lines_cols() lines, cols = curses.LINES, curses.COLS

1 0

bpo-43659: Fix test_curses on AIX (GH-25074)
by miss-islington 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/e9092b221d4951609827e437178a557fd0… commit: e9092b221d4951609827e437178a557fd07353af branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2021-03-29T12:26:37-07:00 summary: bpo-43659: Fix test_curses on AIX (GH-25074) curses.update_lines_cols() is only defined when the curses library provides either resizeterm() or resize_term() functions which are optional and are not provided on AIX. (cherry picked from commit c8b5738810516df5722caf049003e9b319427bec) Co-authored-by: Michael Felt <aixtools(a)users.noreply.github.com> files: M Lib/test/test_curses.py diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index 7dd0695aa7277..c5914698f129f 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -1151,6 +1151,7 @@ def test_issue13051(self): class MiscTests(unittest.TestCase): + @requires_curses_func('update_lines_cols') def test_update_lines_cols(self): curses.update_lines_cols() lines, cols = curses.LINES, curses.COLS

1 0

bpo-43659: Fix test_curses on AIX (GH-25074)
by serhiy-storchaka 29 Mar '21

29 Mar '21

https://github.com/python/cpython/commit/c8b5738810516df5722caf049003e9b319… commit: c8b5738810516df5722caf049003e9b319427bec branch: master author: Michael Felt <aixtools(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2021-03-29T22:06:24+03:00 summary: bpo-43659: Fix test_curses on AIX (GH-25074) curses.update_lines_cols() is only defined when the curses library provides either resizeterm() or resize_term() functions which are optional and are not provided on AIX. files: M Lib/test/test_curses.py diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index 0833c86115ebd..7ce0461ab54f2 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -1174,6 +1174,7 @@ def test_issue13051(self): class MiscTests(unittest.TestCase): + @requires_curses_func('update_lines_cols') def test_update_lines_cols(self): curses.update_lines_cols() lines, cols = curses.LINES, curses.COLS

1 0