Mailman 3 January 2025 - Python-checkins

[3.13] gh-87506: Document that json.load*() can raise UnicodeDecodeError (GH-127355) (#128607)
by erlend-aasland Jan. 8, 2025

Jan. 8, 2025

https://github.com/python/cpython/commit/0cd603689a42504d6724dc9b30218881ce… commit: 0cd603689a42504d6724dc9b30218881ce0a7af2 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2025-01-08T08:48:55Z summary: [3.13] gh-87506: Document that json.load*() can raise UnicodeDecodeError (GH-127355) (#128607) (cherry picked from commit 15372d0112ac7fbbbe313725dd87d5a45af2f6d3) (cherry picked from commit cdfb8bc93a4d8c06d2404ba2d243937ba209438c) Co-authored-by: Srinivas Reddy Thatiparthy (తాటిపర్తి శ్రీనివాస్ రెడ్డి) <thatiparthysreenivas(a)gmail.com> Co-authored-by: Erlend Aasland <erlend(a)python.org> files: M Doc/library/json.rst diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 1a92a44f043277..22636027f9dbaf 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -324,6 +324,10 @@ Basic Usage :raises JSONDecodeError: When the data being deserialized is not a valid JSON document. + :raises UnicodeDecodeError: + When the data being deserialized does not contain + UTF-8, UTF-16 or UTF-32 encoded data. + .. versionchanged:: 3.1 * Added the optional *object_pairs_hook* parameter. @@ -343,15 +347,11 @@ Basic Usage .. function:: loads(s, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) - Deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray` + Identical to :func:`load`, but instead of a file-like object, + deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray` instance containing a JSON document) to a Python object using this :ref:`conversion table <json-to-py-table>`. - The other arguments have the same meaning as in :func:`load`. - - If the data being deserialized is not a valid JSON document, a - :exc:`JSONDecodeError` will be raised. - .. versionchanged:: 3.6 *s* can now be of type :class:`bytes` or :class:`bytearray`. The input encoding should be UTF-8, UTF-16 or UTF-32.

1 0

gh-87506: Amend json.loads() post PR #127355 (#128609)
by erlend-aasland Jan. 8, 2025

Jan. 8, 2025

https://github.com/python/cpython/commit/cdfb8bc93a4d8c06d2404ba2d243937ba2… commit: cdfb8bc93a4d8c06d2404ba2d243937ba209438c branch: main author: Erlend E. Aasland <erlend(a)python.org> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2025-01-08T09:36:44+01:00 summary: gh-87506: Amend json.loads() post PR #127355 (#128609) files: M Doc/library/json.rst diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 41a09b40d7e88b..4e7046d6d8f6ac 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -347,15 +347,11 @@ Basic Usage .. function:: loads(s, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) - Deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray` + Identical to :func:`load`, but instead of a file-like object, + deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray` instance containing a JSON document) to a Python object using this :ref:`conversion table <json-to-py-table>`. - The other arguments have the same meaning as in :func:`load`. - - If the data being deserialized is not a valid JSON document, a - :exc:`JSONDecodeError` will be raised. - .. versionchanged:: 3.6 *s* can now be of type :class:`bytes` or :class:`bytearray`. The input encoding should be UTF-8, UTF-16 or UTF-32.

1 0

gh-87506: Document that json.load*() can raise UnicodeDecodeError (#127355)
by erlend-aasland Jan. 8, 2025

Jan. 8, 2025

https://github.com/python/cpython/commit/15372d0112ac7fbbbe313725dd87d5a45a… commit: 15372d0112ac7fbbbe313725dd87d5a45af2f6d3 branch: main author: Srinivas Reddy Thatiparthy (తాటిపర్తి శ్రీనివాస్ రెడ్డి) <thatiparthysreenivas(a)gmail.com> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2025-01-08T07:35:43Z summary: gh-87506: Document that json.load*() can raise UnicodeDecodeError (#127355) Co-authored-by: Erlend Aasland <erlend(a)python.org> files: M Doc/library/json.rst diff --git a/Doc/library/json.rst b/Doc/library/json.rst index cf516cf3dc1d29..41a09b40d7e88b 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -324,6 +324,10 @@ Basic Usage :raises JSONDecodeError: When the data being deserialized is not a valid JSON document. + :raises UnicodeDecodeError: + When the data being deserialized does not contain + UTF-8, UTF-16 or UTF-32 encoded data. + .. versionchanged:: 3.1 * Added the optional *object_pairs_hook* parameter.

1 0

GH-127809: Fix the JIT's understanding of ** (GH-127844)
by brandtbucher Jan. 8, 2025

Jan. 8, 2025

https://github.com/python/cpython/commit/65ae3d5a73ca3c53a0c6b601dddb8e9b3b… commit: 65ae3d5a73ca3c53a0c6b601dddb8e9b3b6e3f51 branch: main author: Brandt Bucher <brandtbucher(a)microsoft.com> committer: brandtbucher <brandtbucher(a)gmail.com> date: 2025-01-07T17:25:48-08:00 summary: GH-127809: Fix the JIT's understanding of ** (GH-127844) files: A Misc/NEWS.d/next/Core_and_Builtins/2024-12-11-14-32-22.gh-issue-127809.0W8khe.rst M Lib/test/test_capi/test_opt.py M Python/bytecodes.c M Python/executor_cases.c.h M Python/generated_cases.c.h M Python/optimizer_bytecodes.c M Python/optimizer_cases.c.h M Tools/cases_generator/analyzer.py diff --git a/Lib/test/test_capi/test_opt.py b/Lib/test/test_capi/test_opt.py index 4cf9b66170c055..d84702411afe41 100644 --- a/Lib/test/test_capi/test_opt.py +++ b/Lib/test/test_capi/test_opt.py @@ -1,4 +1,5 @@ import contextlib +import itertools import sys import textwrap import unittest @@ -1511,6 +1512,49 @@ def test_jit_error_pops(self): with self.assertRaises(TypeError): {item for item in items} + def test_power_type_depends_on_input_values(self): + template = textwrap.dedent(""" + import _testinternalcapi + + L, R, X, Y = {l}, {r}, {x}, {y} + + def check(actual: complex, expected: complex) -> None: + assert actual == expected, (actual, expected) + assert type(actual) is type(expected), (actual, expected) + + def f(l: complex, r: complex) -> None: + expected_local_local = pow(l, r) + pow(l, r) + expected_const_local = pow(L, r) + pow(L, r) + expected_local_const = pow(l, R) + pow(l, R) + expected_const_const = pow(L, R) + pow(L, R) + for _ in range(_testinternalcapi.TIER2_THRESHOLD): + # Narrow types: + l + l, r + r + # The powers produce results, and the addition is unguarded: + check(l ** r + l ** r, expected_local_local) + check(L ** r + L ** r, expected_const_local) + check(l ** R + l ** R, expected_local_const) + check(L ** R + L ** R, expected_const_const) + + # JIT for one pair of values... + f(L, R) + # ...then run with another: + f(X, Y) + """) + interesting = [ + (1, 1), # int ** int -> int + (1, -1), # int ** int -> float + (1.0, 1), # float ** int -> float + (1, 1.0), # int ** float -> float + (-1, 0.5), # int ** float -> complex + (1.0, 1.0), # float ** float -> float + (-1.0, 0.5), # float ** float -> complex + ] + for (l, r), (x, y) in itertools.product(interesting, repeat=2): + s = template.format(l=l, r=r, x=x, y=y) + with self.subTest(l=l, r=r, x=x, y=y): + script_helper.assert_python_ok("-c", s) + def global_identity(x): return x diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-12-11-14-32-22.gh-issue-127809.0W8khe.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-11-14-32-22.gh-issue-127809.0W8khe.rst new file mode 100644 index 00000000000000..19c8cc6e99c8c5 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-11-14-32-22.gh-issue-127809.0W8khe.rst @@ -0,0 +1,2 @@ +Fix an issue where the experimental JIT may infer an incorrect result type +for exponentiation (``**`` and ``**=``), leading to bugs or crashes. diff --git a/Python/bytecodes.c b/Python/bytecodes.c index ec1cd00962ac0a..8bab4ea16b629b 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -530,6 +530,8 @@ dummy_func( pure op(_BINARY_OP_MULTIPLY_INT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o); @@ -543,6 +545,8 @@ dummy_func( pure op(_BINARY_OP_ADD_INT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o); @@ -556,6 +560,8 @@ dummy_func( pure op(_BINARY_OP_SUBTRACT_INT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o); @@ -593,6 +599,8 @@ dummy_func( pure op(_BINARY_OP_MULTIPLY_FLOAT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = @@ -607,6 +615,8 @@ dummy_func( pure op(_BINARY_OP_ADD_FLOAT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = @@ -621,6 +631,8 @@ dummy_func( pure op(_BINARY_OP_SUBTRACT_FLOAT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = @@ -650,6 +662,8 @@ dummy_func( pure op(_BINARY_OP_ADD_UNICODE, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = PyUnicode_Concat(left_o, right_o); @@ -672,6 +686,8 @@ dummy_func( op(_BINARY_OP_INPLACE_ADD_UNICODE, (left, right --)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); int next_oparg; #if TIER_ONE diff --git a/Python/executor_cases.c.h b/Python/executor_cases.c.h index ac2f69b7e98dc3..e40fa88be89172 100644 --- a/Python/executor_cases.c.h +++ b/Python/executor_cases.c.h @@ -638,6 +638,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -658,6 +660,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -678,6 +682,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -738,6 +744,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval * @@ -759,6 +767,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval + @@ -780,6 +790,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval - @@ -819,6 +831,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = PyUnicode_Concat(left_o, right_o); PyStackRef_CLOSE_SPECIALIZED(left, _PyUnicode_ExactDealloc); @@ -838,6 +852,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); int next_oparg; #if TIER_ONE assert(next_instr->op.code == STORE_FAST); diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index eaa8a563464068..7028ba52faae96 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -80,6 +80,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval + @@ -116,6 +118,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -151,6 +155,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = PyUnicode_Concat(left_o, right_o); PyStackRef_CLOSE_SPECIALIZED(left, _PyUnicode_ExactDealloc); @@ -185,6 +191,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); int next_oparg; #if TIER_ONE assert(next_instr->op.code == STORE_FAST); @@ -247,6 +255,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval * @@ -283,6 +293,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -318,6 +330,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval - @@ -354,6 +368,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); diff --git a/Python/optimizer_bytecodes.c b/Python/optimizer_bytecodes.c index a14d119b7a1dec..86394480f76bb8 100644 --- a/Python/optimizer_bytecodes.c +++ b/Python/optimizer_bytecodes.c @@ -167,23 +167,56 @@ dummy_func(void) { } op(_BINARY_OP, (left, right -- res)) { - PyTypeObject *ltype = sym_get_type(left); - PyTypeObject *rtype = sym_get_type(right); - if (ltype != NULL && (ltype == &PyLong_Type || ltype == &PyFloat_Type) && - rtype != NULL && (rtype == &PyLong_Type || rtype == &PyFloat_Type)) - { - if (oparg != NB_TRUE_DIVIDE && oparg != NB_INPLACE_TRUE_DIVIDE && - ltype == &PyLong_Type && rtype == &PyLong_Type) { - /* If both inputs are ints and the op is not division the result is an int */ - res = sym_new_type(ctx, &PyLong_Type); + bool lhs_int = sym_matches_type(left, &PyLong_Type); + bool rhs_int = sym_matches_type(right, &PyLong_Type); + bool lhs_float = sym_matches_type(left, &PyFloat_Type); + bool rhs_float = sym_matches_type(right, &PyFloat_Type); + if (!((lhs_int || lhs_float) && (rhs_int || rhs_float))) { + // There's something other than an int or float involved: + res = sym_new_unknown(ctx); + } + else if (oparg == NB_POWER || oparg == NB_INPLACE_POWER) { + // This one's fun... the *type* of the result depends on the + // *values* being exponentiated. However, exponents with one + // constant part are reasonably common, so it's probably worth + // trying to infer some simple cases: + // - A: 1 ** 1 -> 1 (int ** int -> int) + // - B: 1 ** -1 -> 1.0 (int ** int -> float) + // - C: 1.0 ** 1 -> 1.0 (float ** int -> float) + // - D: 1 ** 1.0 -> 1.0 (int ** float -> float) + // - E: -1 ** 0.5 ~> 1j (int ** float -> complex) + // - F: 1.0 ** 1.0 -> 1.0 (float ** float -> float) + // - G: -1.0 ** 0.5 ~> 1j (float ** float -> complex) + if (rhs_float) { + // Case D, E, F, or G... can't know without the sign of the LHS + // or whether the RHS is whole, which isn't worth the effort: + res = sym_new_unknown(ctx); } - else { - /* For any other op combining ints/floats the result is a float */ + else if (lhs_float) { + // Case C: res = sym_new_type(ctx, &PyFloat_Type); } + else if (!sym_is_const(right)) { + // Case A or B... can't know without the sign of the RHS: + res = sym_new_unknown(ctx); + } + else if (_PyLong_IsNegative((PyLongObject *)sym_get_const(right))) { + // Case B: + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + // Case A: + res = sym_new_type(ctx, &PyLong_Type); + } + } + else if (oparg == NB_TRUE_DIVIDE || oparg == NB_INPLACE_TRUE_DIVIDE) { + res = sym_new_type(ctx, &PyFloat_Type); + } + else if (lhs_int && rhs_int) { + res = sym_new_type(ctx, &PyLong_Type); } else { - res = sym_new_unknown(ctx); + res = sym_new_type(ctx, &PyFloat_Type); } } diff --git a/Python/optimizer_cases.c.h b/Python/optimizer_cases.c.h index be3e06108aec92..c72ae7b6281e80 100644 --- a/Python/optimizer_cases.c.h +++ b/Python/optimizer_cases.c.h @@ -2307,24 +2307,69 @@ _Py_UopsSymbol *res; right = stack_pointer[-1]; left = stack_pointer[-2]; - PyTypeObject *ltype = sym_get_type(left); - PyTypeObject *rtype = sym_get_type(right); - if (ltype != NULL && (ltype == &PyLong_Type || ltype == &PyFloat_Type) && - rtype != NULL && (rtype == &PyLong_Type || rtype == &PyFloat_Type)) - { - if (oparg != NB_TRUE_DIVIDE && oparg != NB_INPLACE_TRUE_DIVIDE && - ltype == &PyLong_Type && rtype == &PyLong_Type) { - /* If both inputs are ints and the op is not division the result is an int */ - res = sym_new_type(ctx, &PyLong_Type); + bool lhs_int = sym_matches_type(left, &PyLong_Type); + bool rhs_int = sym_matches_type(right, &PyLong_Type); + bool lhs_float = sym_matches_type(left, &PyFloat_Type); + bool rhs_float = sym_matches_type(right, &PyFloat_Type); + if (!((lhs_int || lhs_float) && (rhs_int || rhs_float))) { + // There's something other than an int or float involved: + res = sym_new_unknown(ctx); + } + else { + if (oparg == NB_POWER || oparg == NB_INPLACE_POWER) { + // This one's fun... the *type* of the result depends on the + // *values* being exponentiated. However, exponents with one + // constant part are reasonably common, so it's probably worth + // trying to infer some simple cases: + // - A: 1 ** 1 -> 1 (int ** int -> int) + // - B: 1 ** -1 -> 1.0 (int ** int -> float) + // - C: 1.0 ** 1 -> 1.0 (float ** int -> float) + // - D: 1 ** 1.0 -> 1.0 (int ** float -> float) + // - E: -1 ** 0.5 ~> 1j (int ** float -> complex) + // - F: 1.0 ** 1.0 -> 1.0 (float ** float -> float) + // - G: -1.0 ** 0.5 ~> 1j (float ** float -> complex) + if (rhs_float) { + // Case D, E, F, or G... can't know without the sign of the LHS + // or whether the RHS is whole, which isn't worth the effort: + res = sym_new_unknown(ctx); + } + else { + if (lhs_float) { + // Case C: + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + if (!sym_is_const(right)) { + // Case A or B... can't know without the sign of the RHS: + res = sym_new_unknown(ctx); + } + else { + if (_PyLong_IsNegative((PyLongObject *)sym_get_const(right))) { + // Case B: + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + // Case A: + res = sym_new_type(ctx, &PyLong_Type); + } + } + } + } } else { - /* For any other op combining ints/floats the result is a float */ - res = sym_new_type(ctx, &PyFloat_Type); + if (oparg == NB_TRUE_DIVIDE || oparg == NB_INPLACE_TRUE_DIVIDE) { + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + if (lhs_int && rhs_int) { + res = sym_new_type(ctx, &PyLong_Type); + } + else { + res = sym_new_type(ctx, &PyFloat_Type); + } + } } } - else { - res = sym_new_unknown(ctx); - } stack_pointer[-2] = res; stack_pointer += -1; assert(WITHIN_STACK_BOUNDS()); diff --git a/Tools/cases_generator/analyzer.py b/Tools/cases_generator/analyzer.py index c0a370a936aa94..679beca3ec3a9d 100644 --- a/Tools/cases_generator/analyzer.py +++ b/Tools/cases_generator/analyzer.py @@ -599,6 +599,7 @@ def has_error_without_pop(op: parser.InstDef) -> bool: "_PyLong_CompactValue", "_PyLong_DigitCount", "_PyLong_IsCompact", + "_PyLong_IsNegative", "_PyLong_IsNonNegativeCompact", "_PyLong_IsZero", "_PyLong_Multiply",

1 0

[3.12] gh-123085: Remove double 'import os' added by PR #124021 (#128600)
by jaraco Jan. 8, 2025

Jan. 8, 2025

https://github.com/python/cpython/commit/b46addafb17ad69c8015d1e09a76471bdd… commit: b46addafb17ad69c8015d1e09a76471bdd3a1b0f branch: 3.12 author: Erlend E. Aasland <erlend(a)python.org> committer: jaraco <jaraco(a)jaraco.com> date: 2025-01-07T19:23:17-05:00 summary: [3.12] gh-123085: Remove double 'import os' added by PR #124021 (#128600) gh-123085: Remove double 'import os' added by PR #124021 files: M Lib/test/test_importlib/resources/test_files.py diff --git a/Lib/test/test_importlib/resources/test_files.py b/Lib/test/test_importlib/resources/test_files.py index 489cc95d1d6fa5..fe597d72422f83 100644 --- a/Lib/test/test_importlib/resources/test_files.py +++ b/Lib/test/test_importlib/resources/test_files.py @@ -115,7 +115,6 @@ def test_implicit_files_zip_submodule(self): """ Special test for gh-121735 for Python 3.12. """ - import os import zipfile def create_zip_from_directory(source_dir, zip_filename):

1 0

[3.12] gh-127614: Correctly check for ttyname_r() in configure (#128503) (#128599)
by erlend-aasland Jan. 7, 2025

Jan. 7, 2025

https://github.com/python/cpython/commit/6b3371914c7448373e7dc4bb28020173b1… commit: 6b3371914c7448373e7dc4bb28020173b10dbe02 branch: 3.12 author: Erlend E. Aasland <erlend(a)python.org> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2025-01-07T22:05:42Z summary: [3.12] gh-127614: Correctly check for ttyname_r() in configure (#128503) (#128599) (cherry picked from commit e08b28235a863323ca3a7e444776bb7803e77caf) PR #14868 replaced the ttyname() call with ttyname_r(), but the old check remained. files: M Modules/clinic/posixmodule.c.h M Modules/posixmodule.c M configure M configure.ac M pyconfig.h.in diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index a33461dc5600dd..57121e1e8903a5 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -304,7 +304,7 @@ os_access(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *k return return_value; } -#if defined(HAVE_TTYNAME) +#if defined(HAVE_TTYNAME_R) PyDoc_STRVAR(os_ttyname__doc__, "ttyname($module, fd, /)\n" @@ -337,7 +337,7 @@ os_ttyname(PyObject *module, PyObject *arg) return return_value; } -#endif /* defined(HAVE_TTYNAME) */ +#endif /* defined(HAVE_TTYNAME_R) */ #if defined(HAVE_CTERMID) @@ -12032,4 +12032,4 @@ os_waitstatus_to_exitcode(PyObject *module, PyObject *const *args, Py_ssize_t na #ifndef OS_WAITSTATUS_TO_EXITCODE_METHODDEF #define OS_WAITSTATUS_TO_EXITCODE_METHODDEF #endif /* !defined(OS_WAITSTATUS_TO_EXITCODE_METHODDEF) */ -/*[clinic end generated code: output=6d34c4564aca7725 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=efe45f93561db8d3 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 82078a569881a0..deabc103056642 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3237,7 +3237,7 @@ os_access_impl(PyObject *module, path_t *path, int mode, int dir_fd, #endif -#ifdef HAVE_TTYNAME +#ifdef HAVE_TTYNAME_R /*[clinic input] os.ttyname diff --git a/configure b/configure index 32c9600b2e8cac..89edc42f45c361 100755 --- a/configure +++ b/configure @@ -18553,10 +18553,10 @@ then : printf "%s\n" "#define HAVE_TRUNCATE 1" >>confdefs.h fi -ac_fn_c_check_func "$LINENO" "ttyname" "ac_cv_func_ttyname" -if test "x$ac_cv_func_ttyname" = xyes +ac_fn_c_check_func "$LINENO" "ttyname_r" "ac_cv_func_ttyname_r" +if test "x$ac_cv_func_ttyname_r" = xyes then : - printf "%s\n" "#define HAVE_TTYNAME 1" >>confdefs.h + printf "%s\n" "#define HAVE_TTYNAME_R 1" >>confdefs.h fi ac_fn_c_check_func "$LINENO" "umask" "ac_cv_func_umask" diff --git a/configure.ac b/configure.ac index 0f60c849965719..1a02d19f1b2a3f 100644 --- a/configure.ac +++ b/configure.ac @@ -4941,7 +4941,7 @@ AC_CHECK_FUNCS([ \ sigfillset siginterrupt sigpending sigrelse sigtimedwait sigwait \ sigwaitinfo snprintf splice strftime strlcpy strsignal symlinkat sync \ sysconf system tcgetpgrp tcsetpgrp tempnam timegm times tmpfile \ - tmpnam tmpnam_r truncate ttyname umask uname unlinkat utimensat utimes vfork \ + tmpnam tmpnam_r truncate ttyname_r umask uname unlinkat utimensat utimes vfork \ wait wait3 wait4 waitid waitpid wcscoll wcsftime wcsxfrm wmemcmp writev \ ]) diff --git a/pyconfig.h.in b/pyconfig.h.in index 167c6a11411aa6..df4d29fe5497a4 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1435,8 +1435,8 @@ /* Define to 1 if you have the `truncate' function. */ #undef HAVE_TRUNCATE -/* Define to 1 if you have the `ttyname' function. */ -#undef HAVE_TTYNAME +/* Define to 1 if you have the `ttyname_r' function. */ +#undef HAVE_TTYNAME_R /* Define to 1 if you don't have `tm_zone' but do have the external array `tzname'. */

1 0

gh-127614: Correctly check for ttyname_r() in configure (#128503)
by erlend-aasland Jan. 7, 2025

Jan. 7, 2025

https://github.com/python/cpython/commit/e08b28235a863323ca3a7e444776bb7803… commit: e08b28235a863323ca3a7e444776bb7803e77caf branch: main author: Erlend E. Aasland <erlend(a)python.org> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2025-01-07T22:42:36+01:00 summary: gh-127614: Correctly check for ttyname_r() in configure (#128503) PR #14868 replaced the ttyname() call with ttyname_r(), but the old check remained. files: M Modules/clinic/posixmodule.c.h M Modules/posixmodule.c M configure M configure.ac M pyconfig.h.in diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 554299b8598299..4e6c5b068c42b7 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -309,7 +309,7 @@ os_access(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *k return return_value; } -#if defined(HAVE_TTYNAME) +#if defined(HAVE_TTYNAME_R) PyDoc_STRVAR(os_ttyname__doc__, "ttyname($module, fd, /)\n" @@ -342,7 +342,7 @@ os_ttyname(PyObject *module, PyObject *arg) return return_value; } -#endif /* defined(HAVE_TTYNAME) */ +#endif /* defined(HAVE_TTYNAME_R) */ #if defined(HAVE_CTERMID) @@ -13140,4 +13140,4 @@ os__emscripten_debugger(PyObject *module, PyObject *Py_UNUSED(ignored)) #ifndef OS__EMSCRIPTEN_DEBUGGER_METHODDEF #define OS__EMSCRIPTEN_DEBUGGER_METHODDEF #endif /* !defined(OS__EMSCRIPTEN_DEBUGGER_METHODDEF) */ -/*[clinic end generated code: output=9c2ca1dbf986c62c input=a9049054013a1b77]*/ +/*[clinic end generated code: output=39b69b279fd637f7 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 151d469983fafb..bb8d698bfed375 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3347,7 +3347,7 @@ os_access_impl(PyObject *module, path_t *path, int mode, int dir_fd, #endif -#ifdef HAVE_TTYNAME +#ifdef HAVE_TTYNAME_R /*[clinic input] os.ttyname diff --git a/configure b/configure index 08c10ff52ce8d5..6e1b393a3ece68 100755 --- a/configure +++ b/configure @@ -19986,10 +19986,10 @@ then : printf "%s\n" "#define HAVE_TRUNCATE 1" >>confdefs.h fi -ac_fn_c_check_func "$LINENO" "ttyname" "ac_cv_func_ttyname" -if test "x$ac_cv_func_ttyname" = xyes +ac_fn_c_check_func "$LINENO" "ttyname_r" "ac_cv_func_ttyname_r" +if test "x$ac_cv_func_ttyname_r" = xyes then : - printf "%s\n" "#define HAVE_TTYNAME 1" >>confdefs.h + printf "%s\n" "#define HAVE_TTYNAME_R 1" >>confdefs.h fi ac_fn_c_check_func "$LINENO" "umask" "ac_cv_func_umask" diff --git a/configure.ac b/configure.ac index fb1dd77be167b9..6d44be8959865d 100644 --- a/configure.ac +++ b/configure.ac @@ -5152,7 +5152,7 @@ AC_CHECK_FUNCS([ \ sigfillset siginterrupt sigpending sigrelse sigtimedwait sigwait \ sigwaitinfo snprintf splice strftime strlcpy strsignal symlinkat sync \ sysconf tcgetpgrp tcsetpgrp tempnam timegm times tmpfile \ - tmpnam tmpnam_r truncate ttyname umask uname unlinkat unlockpt utimensat utimes vfork \ + tmpnam tmpnam_r truncate ttyname_r umask uname unlinkat unlockpt utimensat utimes vfork \ wait wait3 wait4 waitid waitpid wcscoll wcsftime wcsxfrm wmemcmp writev \ ]) diff --git a/pyconfig.h.in b/pyconfig.h.in index d862966b7de4c8..874b98dc96585a 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1506,8 +1506,8 @@ /* Define to 1 if you have the 'truncate' function. */ #undef HAVE_TRUNCATE -/* Define to 1 if you have the 'ttyname' function. */ -#undef HAVE_TTYNAME +/* Define to 1 if you have the 'ttyname_r' function. */ +#undef HAVE_TTYNAME_R /* Define to 1 if you don't have 'tm_zone' but do have the external array 'tzname'. */

1 0

[3.12] Docs: mark up json.load() using parameter list (GH-128488) (#128597)
by erlend-aasland Jan. 7, 2025

Jan. 7, 2025

https://github.com/python/cpython/commit/82b061d71157226e127efe037cb3ae1be8… commit: 82b061d71157226e127efe037cb3ae1be89a09c3 branch: 3.12 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2025-01-07T21:14:50Z summary: [3.12] Docs: mark up json.load() using parameter list (GH-128488) (#128597) (cherry picked from commit a21e31ec5417374497b0f969ee2b3dc84a0944ff) Co-authored-by: Erlend E. Aasland <erlend(a)python.org> files: M Doc/library/json.rst diff --git a/Doc/library/json.rst b/Doc/library/json.rst index eff8808a440528..88730d0cd25b8d 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -247,36 +247,82 @@ Basic Usage the original one. That is, ``loads(dumps(x)) != x`` if x has non-string keys. -.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) +.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, \ + parse_int=None, parse_constant=None, \ + object_pairs_hook=None, **kw) - Deserialize *fp* (a ``.read()``-supporting :term:`text file` or - :term:`binary file` containing a JSON document) to a Python object using - this :ref:`conversion table <json-to-py-table>`. + Deserialize *fp* to a Python object + using the :ref:`JSON-to-Python conversion table <json-to-py-table>`. - *object_hook* is an optional function that will be called with the result of - any object literal decoded (a :class:`dict`). The return value of - *object_hook* will be used instead of the :class:`dict`. This feature can - be used to implement custom decoders (e.g. `JSON-RPC - <https://www.jsonrpc.org>`_ class hinting). + :param fp: + A ``.read()``-supporting :term:`text file` or :term:`binary file` + containing the JSON document to be deserialized. + :type fp: :term:`file-like object` - *object_pairs_hook* is an optional function that will be called with the - result of any object literal decoded with an ordered list of pairs. The - return value of *object_pairs_hook* will be used instead of the - :class:`dict`. This feature can be used to implement custom decoders. If - *object_hook* is also defined, the *object_pairs_hook* takes priority. + :param cls: + If set, a custom JSON decoder. + Additional keyword arguments to :func:`!load` + will be passed to the constructor of *cls*. + If ``None`` (the default), :class:`!JSONDecoder` is used. + :type cls: a :class:`JSONDecoder` subclass + + :param object_hook: + If set, a function that is called with the result of + any object literal decoded (a :class:`dict`). + The return value of this function will be used + instead of the :class:`dict`. + This feature can be used to implement custom decoders, + for example `JSON-RPC <https://www.jsonrpc.org>`_ class hinting. + Default ``None``. + :type object_hook: :term:`callable` | None + + :param object_pairs_hook: + If set, a function that is called with the result of + any object literal decoded with an ordered list of pairs. + The return value of this function will be used + instead of the :class:`dict`. + This feature can be used to implement custom decoders. + If *object_hook* is also set, *object_pairs_hook* takes priority. + Default ``None``. + :type object_pairs_hook: :term:`callable` | None + + :param parse_float: + If set, a function that is called with + the string of every JSON float to be decoded. + If ``None`` (the default), it is equivalent to ``float(num_str)``. + This can be used to parse JSON floats into custom datatypes, + for example :class:`decimal.Decimal`. + :type parse_float: :term:`callable` | None + + :param parse_int: + If set, a function that is called with + the string of every JSON int to be decoded. + If ``None`` (the default), it is equivalent to ``int(num_str)``. + This can be used to parse JSON integers into custom datatypes, + for example :class:`float`. + :type parse_int: :term:`callable` | None + + :param parse_constant: + If set, a function that is called with one of the following strings: + ``'-Infinity'``, ``'Infinity'``, or ``'NaN'``. + This can be used to raise an exception + if invalid JSON numbers are encountered. + Default ``None``. + :type parse_constant: :term:`callable` | None + + :raises JSONDecodeError: + When the data being deserialized is not a valid JSON document. .. versionchanged:: 3.1 - Added support for *object_pairs_hook*. - *parse_float* is an optional function that will be called with the string of - every JSON float to be decoded. By default, this is equivalent to - ``float(num_str)``. This can be used to use another datatype or parser for - JSON floats (e.g. :class:`decimal.Decimal`). + * Added the optional *object_pairs_hook* parameter. + * *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. - *parse_int* is an optional function that will be called with the string of - every JSON int to be decoded. By default, this is equivalent to - ``int(num_str)``. This can be used to use another datatype or parser for - JSON integers (e.g. :class:`float`). + .. versionchanged:: 3.6 + + * All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. + * *fp* can now be a :term:`binary file`. + The input encoding should be UTF-8, UTF-16 or UTF-32. .. versionchanged:: 3.11 The default *parse_int* of :func:`int` now limits the maximum length of @@ -284,27 +330,6 @@ Basic Usage conversion length limitation <int_max_str_digits>` to help avoid denial of service attacks. - *parse_constant* is an optional function that will be called with one of the - following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. This can be - used to raise an exception if invalid JSON numbers are encountered. - - .. versionchanged:: 3.1 - *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. - - To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` - kwarg; otherwise :class:`JSONDecoder` is used. Additional keyword arguments - will be passed to the constructor of the class. - - If the data being deserialized is not a valid JSON document, a - :exc:`JSONDecodeError` will be raised. - - .. versionchanged:: 3.6 - All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. - - .. versionchanged:: 3.6 - *fp* can now be a :term:`binary file`. The input encoding should be - UTF-8, UTF-16 or UTF-32. - .. function:: loads(s, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) Deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray`

1 0

[3.13] Docs: mark up json.load() using parameter list (GH-128488) (#128596)
by erlend-aasland Jan. 7, 2025

Jan. 7, 2025

https://github.com/python/cpython/commit/984cdb788ec1be6d5e2c1a7e651ee37d53… commit: 984cdb788ec1be6d5e2c1a7e651ee37d53ad05ed branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2025-01-07T21:13:25Z summary: [3.13] Docs: mark up json.load() using parameter list (GH-128488) (#128596) (cherry picked from commit a21e31ec5417374497b0f969ee2b3dc84a0944ff) Co-authored-by: Erlend E. Aasland <erlend(a)python.org> files: M Doc/library/json.rst diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 21f36cb38f1193..1a92a44f043277 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -258,36 +258,82 @@ Basic Usage the original one. That is, ``loads(dumps(x)) != x`` if x has non-string keys. -.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) +.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, \ + parse_int=None, parse_constant=None, \ + object_pairs_hook=None, **kw) - Deserialize *fp* (a ``.read()``-supporting :term:`text file` or - :term:`binary file` containing a JSON document) to a Python object using - this :ref:`conversion table <json-to-py-table>`. + Deserialize *fp* to a Python object + using the :ref:`JSON-to-Python conversion table <json-to-py-table>`. - *object_hook* is an optional function that will be called with the result of - any object literal decoded (a :class:`dict`). The return value of - *object_hook* will be used instead of the :class:`dict`. This feature can - be used to implement custom decoders (e.g. `JSON-RPC - <https://www.jsonrpc.org>`_ class hinting). + :param fp: + A ``.read()``-supporting :term:`text file` or :term:`binary file` + containing the JSON document to be deserialized. + :type fp: :term:`file-like object` - *object_pairs_hook* is an optional function that will be called with the - result of any object literal decoded with an ordered list of pairs. The - return value of *object_pairs_hook* will be used instead of the - :class:`dict`. This feature can be used to implement custom decoders. If - *object_hook* is also defined, the *object_pairs_hook* takes priority. + :param cls: + If set, a custom JSON decoder. + Additional keyword arguments to :func:`!load` + will be passed to the constructor of *cls*. + If ``None`` (the default), :class:`!JSONDecoder` is used. + :type cls: a :class:`JSONDecoder` subclass + + :param object_hook: + If set, a function that is called with the result of + any object literal decoded (a :class:`dict`). + The return value of this function will be used + instead of the :class:`dict`. + This feature can be used to implement custom decoders, + for example `JSON-RPC <https://www.jsonrpc.org>`_ class hinting. + Default ``None``. + :type object_hook: :term:`callable` | None + + :param object_pairs_hook: + If set, a function that is called with the result of + any object literal decoded with an ordered list of pairs. + The return value of this function will be used + instead of the :class:`dict`. + This feature can be used to implement custom decoders. + If *object_hook* is also set, *object_pairs_hook* takes priority. + Default ``None``. + :type object_pairs_hook: :term:`callable` | None + + :param parse_float: + If set, a function that is called with + the string of every JSON float to be decoded. + If ``None`` (the default), it is equivalent to ``float(num_str)``. + This can be used to parse JSON floats into custom datatypes, + for example :class:`decimal.Decimal`. + :type parse_float: :term:`callable` | None + + :param parse_int: + If set, a function that is called with + the string of every JSON int to be decoded. + If ``None`` (the default), it is equivalent to ``int(num_str)``. + This can be used to parse JSON integers into custom datatypes, + for example :class:`float`. + :type parse_int: :term:`callable` | None + + :param parse_constant: + If set, a function that is called with one of the following strings: + ``'-Infinity'``, ``'Infinity'``, or ``'NaN'``. + This can be used to raise an exception + if invalid JSON numbers are encountered. + Default ``None``. + :type parse_constant: :term:`callable` | None + + :raises JSONDecodeError: + When the data being deserialized is not a valid JSON document. .. versionchanged:: 3.1 - Added support for *object_pairs_hook*. - *parse_float* is an optional function that will be called with the string of - every JSON float to be decoded. By default, this is equivalent to - ``float(num_str)``. This can be used to use another datatype or parser for - JSON floats (e.g. :class:`decimal.Decimal`). + * Added the optional *object_pairs_hook* parameter. + * *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. - *parse_int* is an optional function that will be called with the string of - every JSON int to be decoded. By default, this is equivalent to - ``int(num_str)``. This can be used to use another datatype or parser for - JSON integers (e.g. :class:`float`). + .. versionchanged:: 3.6 + + * All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. + * *fp* can now be a :term:`binary file`. + The input encoding should be UTF-8, UTF-16 or UTF-32. .. versionchanged:: 3.11 The default *parse_int* of :func:`int` now limits the maximum length of @@ -295,27 +341,6 @@ Basic Usage conversion length limitation <int_max_str_digits>` to help avoid denial of service attacks. - *parse_constant* is an optional function that will be called with one of the - following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. This can be - used to raise an exception if invalid JSON numbers are encountered. - - .. versionchanged:: 3.1 - *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. - - To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` - kwarg; otherwise :class:`JSONDecoder` is used. Additional keyword arguments - will be passed to the constructor of the class. - - If the data being deserialized is not a valid JSON document, a - :exc:`JSONDecodeError` will be raised. - - .. versionchanged:: 3.6 - All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. - - .. versionchanged:: 3.6 - *fp* can now be a :term:`binary file`. The input encoding should be - UTF-8, UTF-16 or UTF-32. - .. function:: loads(s, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) Deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray`

1 0

gh-127350: Add more tests for Py_fopen() (GH-128587)
by serhiy-storchaka Jan. 7, 2025

Jan. 7, 2025

https://github.com/python/cpython/commit/07e6aa2efc5b2c7ecbbdb41636013c64e3… commit: 07e6aa2efc5b2c7ecbbdb41636013c64e335a2ba branch: main author: Serhiy Storchaka <storchaka(a)gmail.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2025-01-07T21:13:08Z summary: gh-127350: Add more tests for Py_fopen() (GH-128587) files: M Lib/test/test_capi/test_file.py M Modules/_testcapi/clinic/file.c.h M Modules/_testcapi/file.c diff --git a/Lib/test/test_capi/test_file.py b/Lib/test/test_capi/test_file.py index 8a08a0a93eb8b7..a67a5121c4588b 100644 --- a/Lib/test/test_capi/test_file.py +++ b/Lib/test/test_capi/test_file.py @@ -5,6 +5,8 @@ _testcapi = import_helper.import_module('_testcapi') +NULL = None + class CAPIFileTest(unittest.TestCase): def test_py_fopen(self): @@ -25,7 +27,9 @@ def test_py_fopen(self): os_helper.TESTFN, os.fsencode(os_helper.TESTFN), ] - # TESTFN_UNDECODABLE cannot be used to create a file on macOS/WASI. + if os_helper.TESTFN_UNDECODABLE is not None: + filenames.append(os_helper.TESTFN_UNDECODABLE) + filenames.append(os.fsdecode(os_helper.TESTFN_UNDECODABLE)) if os_helper.TESTFN_UNENCODABLE is not None: filenames.append(os_helper.TESTFN_UNENCODABLE) for filename in filenames: @@ -33,7 +37,12 @@ def test_py_fopen(self): try: with open(filename, "wb") as fp: fp.write(source) - + except OSError: + # TESTFN_UNDECODABLE cannot be used to create a file + # on macOS/WASI. + filename = None + continue + try: data = _testcapi.py_fopen(filename, "rb") self.assertEqual(data, source[:256]) finally: @@ -47,7 +56,14 @@ def test_py_fopen(self): # non-ASCII mode failing with "Invalid argument" with self.assertRaises(OSError): - _testcapi.py_fopen(__file__, "\xe9") + _testcapi.py_fopen(__file__, b"\xc2\x80") + with self.assertRaises(OSError): + # \x98 is invalid in cp1250, cp1251, cp1257 + # \x9d is invalid in cp1252-cp1255, cp1258 + _testcapi.py_fopen(__file__, b"\xc2\x98\xc2\x9d") + # UnicodeDecodeError can come from the audit hook code + with self.assertRaises((UnicodeDecodeError, OSError)): + _testcapi.py_fopen(__file__, b"\x98\x9d") # invalid filename type for invalid_type in (123, object()): @@ -60,7 +76,8 @@ def test_py_fopen(self): # On Windows, the file mode is limited to 10 characters _testcapi.py_fopen(__file__, "rt+, ccs=UTF-8") - # CRASHES py_fopen(__file__, None) + # CRASHES _testcapi.py_fopen(NULL, 'rb') + # CRASHES _testcapi.py_fopen(__file__, NULL) if __name__ == "__main__": diff --git a/Modules/_testcapi/clinic/file.c.h b/Modules/_testcapi/clinic/file.c.h index 2ca21fffcef680..fddbf48071bd3b 100644 --- a/Modules/_testcapi/clinic/file.c.h +++ b/Modules/_testcapi/clinic/file.c.h @@ -14,7 +14,8 @@ PyDoc_STRVAR(_testcapi_py_fopen__doc__, {"py_fopen", _PyCFunction_CAST(_testcapi_py_fopen), METH_FASTCALL, _testcapi_py_fopen__doc__}, static PyObject * -_testcapi_py_fopen_impl(PyObject *module, PyObject *path, const char *mode); +_testcapi_py_fopen_impl(PyObject *module, PyObject *path, const char *mode, + Py_ssize_t mode_length); static PyObject * _testcapi_py_fopen(PyObject *module, PyObject *const *args, Py_ssize_t nargs) @@ -22,27 +23,15 @@ _testcapi_py_fopen(PyObject *module, PyObject *const *args, Py_ssize_t nargs) PyObject *return_value = NULL; PyObject *path; const char *mode; - - if (!_PyArg_CheckPositional("py_fopen", nargs, 2, 2)) { - goto exit; - } - path = args[0]; - if (!PyUnicode_Check(args[1])) { - _PyArg_BadArgument("py_fopen", "argument 2", "str", args[1]); - goto exit; - } Py_ssize_t mode_length; - mode = PyUnicode_AsUTF8AndSize(args[1], &mode_length); - if (mode == NULL) { - goto exit; - } - if (strlen(mode) != (size_t)mode_length) { - PyErr_SetString(PyExc_ValueError, "embedded null character"); + + if (!_PyArg_ParseStack(args, nargs, "Oz#:py_fopen", + &path, &mode, &mode_length)) { goto exit; } - return_value = _testcapi_py_fopen_impl(module, path, mode); + return_value = _testcapi_py_fopen_impl(module, path, mode, mode_length); exit: return return_value; } -/*[clinic end generated code: output=c9fe964c3e5a0c32 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=c4dc92400306c3eb input=a9049054013a1b77]*/ diff --git a/Modules/_testcapi/file.c b/Modules/_testcapi/file.c index 4bad43010fd440..d15173fc7959e5 100644 --- a/Modules/_testcapi/file.c +++ b/Modules/_testcapi/file.c @@ -14,16 +14,18 @@ module _testcapi _testcapi.py_fopen path: object - mode: str + mode: str(zeroes=True, accept={robuffer, str, NoneType}) / Call Py_fopen(), fread(256) and Py_fclose(). Return read bytes. [clinic start generated code]*/ static PyObject * -_testcapi_py_fopen_impl(PyObject *module, PyObject *path, const char *mode) -/*[clinic end generated code: output=5a900af000f759de input=d7e7b8f0fd151953]*/ +_testcapi_py_fopen_impl(PyObject *module, PyObject *path, const char *mode, + Py_ssize_t mode_length) +/*[clinic end generated code: output=69840d0cfd8b7fbb input=f3a579dd7eb60926]*/ { + NULLABLE(path); FILE *fp = Py_fopen(path, mode); if (fp == NULL) { return NULL;

1 0