Mailman 3 January 2025 - Python-checkins

gh-128691: Use deferred reference counting on `_thread._local` (#128693)
by colesbury Jan. 10, 2025

Jan. 10, 2025

https://github.com/python/cpython/commit/c1417487e98e270d614965ed78ff943904… commit: c1417487e98e270d614965ed78ff9439044b65a6 branch: main author: Sam Gross <colesbury(a)gmail.com> committer: colesbury <colesbury(a)gmail.com> date: 2025-01-10T00:59:10Z summary: gh-128691: Use deferred reference counting on `_thread._local` (#128693) This change, along with the LOAD_ATTR specializations, makes the "thread_local_read" micro benchmark in Tools/ftscalingbench/ftscalingbench.py scale well to multiple threads. files: M Modules/_threadmodule.c diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index 2cbdfeb09b95ae..d19ae326bd6b48 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -1414,6 +1414,10 @@ local_new(PyTypeObject *type, PyObject *args, PyObject *kw) return NULL; } + // gh-128691: Use deferred reference counting for thread-locals to avoid + // contention on the shared object. + _PyObject_SetDeferredRefcount((PyObject *)self); + self->args = Py_XNewRef(args); self->kw = Py_XNewRef(kw);

1 0

[3.13] gh-127196: Fix crash in `_interpreters`, when `shared` had invalid encodings (GH-127220) (#128689)
by sobolevn Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/0d2b9abd18ff4fc6d2fbf1b398b411ed3b… commit: 0d2b9abd18ff4fc6d2fbf1b398b411ed3b76266c branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: sobolevn <mail(a)sobolevn.me> date: 2025-01-09T20:10:08Z summary: [3.13] gh-127196: Fix crash in `_interpreters`, when `shared` had invalid encodings (GH-127220) (#128689) gh-127196: Fix crash in `_interpreters`, when `shared` had invalid encodings (GH-127220) (cherry picked from commit 087bb48acac997c06e69dae25bae2dd75194b980) Co-authored-by: sobolevn <mail(a)sobolevn.me> files: A Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst M Lib/test/test__interpreters.py M Modules/_interpretersmodule.c diff --git a/Lib/test/test__interpreters.py b/Lib/test/test__interpreters.py index 533120a3221987..8f32a446572d5a 100644 --- a/Lib/test/test__interpreters.py +++ b/Lib/test/test__interpreters.py @@ -557,7 +557,7 @@ def setUp(self): self.id = _interpreters.create() def test_signatures(self): - # for method in ['exec', 'run_string', 'run_func']: + # See https://github.com/python/cpython/issues/126654 msg = "expected 'shared' to be a dict" with self.assertRaisesRegex(TypeError, msg): _interpreters.exec(self.id, 'a', 1) @@ -568,6 +568,17 @@ def test_signatures(self): with self.assertRaisesRegex(TypeError, msg): _interpreters.run_func(self.id, lambda: None, shared=1) + def test_invalid_shared_encoding(self): + # See https://github.com/python/cpython/issues/127196 + bad_shared = {"\uD82A": 0} + msg = 'surrogates not allowed' + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.exec(self.id, 'a', shared=bad_shared) + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.run_string(self.id, 'a', shared=bad_shared) + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.run_func(self.id, lambda: None, shared=bad_shared) + class RunStringTests(TestBase): diff --git a/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst b/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst new file mode 100644 index 00000000000000..471f64d185deab --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst @@ -0,0 +1,2 @@ +Fix crash when dict with keys in invalid encoding were passed to several +functions in ``_interpreters`` module. diff --git a/Modules/_interpretersmodule.c b/Modules/_interpretersmodule.c index 706316234c7d24..c968b33939c420 100644 --- a/Modules/_interpretersmodule.c +++ b/Modules/_interpretersmodule.c @@ -462,7 +462,12 @@ _run_in_interpreter(PyInterpreterState *interp, // Prep and switch interpreters. if (_PyXI_Enter(&session, interp, shareables) < 0) { - assert(!PyErr_Occurred()); + if (PyErr_Occurred()) { + // If an error occured at this step, it means that interp + // was not prepared and switched. + return -1; + } + // Now, apply the error from another interpreter: PyObject *excinfo = _PyXI_ApplyError(session.error); if (excinfo != NULL) { *p_excinfo = excinfo;

1 0

gh-127196: Fix crash in `_interpreters`, when `shared` had invalid encodings (#127220)
by sobolevn Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/087bb48acac997c06e69dae25bae2dd751… commit: 087bb48acac997c06e69dae25bae2dd75194b980 branch: main author: sobolevn <mail(a)sobolevn.me> committer: sobolevn <mail(a)sobolevn.me> date: 2025-01-09T19:45:35Z summary: gh-127196: Fix crash in `_interpreters`, when `shared` had invalid encodings (#127220) files: A Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst M Lib/test/test__interpreters.py M Modules/_interpretersmodule.c diff --git a/Lib/test/test__interpreters.py b/Lib/test/test__interpreters.py index bf3165e2341949..fd444f1f06ce48 100644 --- a/Lib/test/test__interpreters.py +++ b/Lib/test/test__interpreters.py @@ -557,7 +557,7 @@ def setUp(self): self.id = _interpreters.create() def test_signatures(self): - # for method in ['exec', 'run_string', 'run_func']: + # See https://github.com/python/cpython/issues/126654 msg = "expected 'shared' to be a dict" with self.assertRaisesRegex(TypeError, msg): _interpreters.exec(self.id, 'a', 1) @@ -568,6 +568,17 @@ def test_signatures(self): with self.assertRaisesRegex(TypeError, msg): _interpreters.run_func(self.id, lambda: None, shared=1) + def test_invalid_shared_encoding(self): + # See https://github.com/python/cpython/issues/127196 + bad_shared = {"\uD82A": 0} + msg = 'surrogates not allowed' + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.exec(self.id, 'a', shared=bad_shared) + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.run_string(self.id, 'a', shared=bad_shared) + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.run_func(self.id, lambda: None, shared=bad_shared) + class RunStringTests(TestBase): diff --git a/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst b/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst new file mode 100644 index 00000000000000..471f64d185deab --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst @@ -0,0 +1,2 @@ +Fix crash when dict with keys in invalid encoding were passed to several +functions in ``_interpreters`` module. diff --git a/Modules/_interpretersmodule.c b/Modules/_interpretersmodule.c index a36823c4bb982b..fcd0baf696f943 100644 --- a/Modules/_interpretersmodule.c +++ b/Modules/_interpretersmodule.c @@ -459,7 +459,12 @@ _run_in_interpreter(PyInterpreterState *interp, // Prep and switch interpreters. if (_PyXI_Enter(&session, interp, shareables) < 0) { - assert(!PyErr_Occurred()); + if (PyErr_Occurred()) { + // If an error occured at this step, it means that interp + // was not prepared and switched. + return -1; + } + // Now, apply the error from another interpreter: PyObject *excinfo = _PyXI_ApplyError(session.error); if (excinfo != NULL) { *p_excinfo = excinfo;

1 0

gh-128650: Fix incorrect statement in partial documentation (gh-128651)
by rhettinger Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/8af57810946c216b3e18c94c8f0ee3c0c9… commit: 8af57810946c216b3e18c94c8f0ee3c0c96566a9 branch: main author: dgpb <3577712+dg-pb(a)users.noreply.github.com> committer: rhettinger <rhettinger(a)users.noreply.github.com> date: 2025-01-09T13:32:00-06:00 summary: gh-128650: Fix incorrect statement in partial documentation (gh-128651) files: M Doc/library/functools.rst diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index 69d9d81c848124..8ad5f48c9e5286 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -368,8 +368,8 @@ The :mod:`functools` module defines the following functions: If :data:`Placeholder` sentinels are present in *args*, they will be filled first when :func:`!partial` is called. This makes it possible to pre-fill any positional - argument with a call to :func:`!partial`; without :data:`!Placeholder`, only the - first positional argument can be pre-filled. + argument with a call to :func:`!partial`; without :data:`!Placeholder`, + only the chosen number of leading positional arguments can be pre-filled. If any :data:`!Placeholder` sentinels are present, all must be filled at call time:

1 0

gh-128002: fix `asyncio.all_tasks` against concurrent deallocations of tasks (#128541)
by kumaraditya303 Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/7dc41ad6a7826ffc675f088972de966249… commit: 7dc41ad6a7826ffc675f088972de96624917696e branch: main author: Kumar Aditya <kumaraditya(a)python.org> committer: kumaraditya303 <kumaraditya(a)python.org> date: 2025-01-09T21:26:00+05:30 summary: gh-128002: fix `asyncio.all_tasks` against concurrent deallocations of tasks (#128541) files: M Include/internal/pycore_object.h M Modules/_asynciomodule.c diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h index d7d68f938a9f0a..e26cb7673f939c 100644 --- a/Include/internal/pycore_object.h +++ b/Include/internal/pycore_object.h @@ -120,7 +120,7 @@ PyAPI_FUNC(void) _Py_NO_RETURN _Py_FatalRefcountErrorFunc( PyAPI_DATA(Py_ssize_t) _Py_RefTotal; extern void _Py_AddRefTotal(PyThreadState *, Py_ssize_t); -extern void _Py_IncRefTotal(PyThreadState *); +extern PyAPI_FUNC(void) _Py_IncRefTotal(PyThreadState *); extern void _Py_DecRefTotal(PyThreadState *); # define _Py_DEC_REFTOTAL(interp) \ diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index b8b184af04a7cb..48f0ef95934fa4 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -3772,11 +3772,20 @@ _asyncio_all_tasks_impl(PyObject *module, PyObject *loop) llist_for_each_safe(node, &state->asyncio_tasks_head) { TaskObj *task = llist_data(node, TaskObj, task_node); - if (PyList_Append(tasks, (PyObject *)task) < 0) { - Py_DECREF(tasks); - Py_DECREF(loop); - err = 1; - break; + // The linked list holds borrowed references to task + // as such it is possible that the task is concurrently + // deallocated while added to this list. + // To protect against concurrent deallocations, + // we first try to incref the task which would fail + // if it is concurrently getting deallocated in another thread, + // otherwise it gets added to the list. + if (_Py_TryIncref((PyObject *)task)) { + if (_PyList_AppendTakeRef((PyListObject *)tasks, (PyObject *)task) < 0) { + Py_DECREF(tasks); + Py_DECREF(loop); + err = 1; + break; + } } } ASYNCIO_STATE_UNLOCK(state);

1 0

gh-128661: Fix `typing.evaluate_forward_ref` not showing deprecation (#128663)
by sobolevn Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/b725297cee9e5608b709f3c7291d974c97… commit: b725297cee9e5608b709f3c7291d974c97f68fff branch: main author: sobolevn <mail(a)sobolevn.me> committer: sobolevn <mail(a)sobolevn.me> date: 2025-01-09T18:15:13+03:00 summary: gh-128661: Fix `typing.evaluate_forward_ref` not showing deprecation (#128663) gh-128661: Fix `typing.evaluate_forward_ref` not showing deprecataion files: A Misc/NEWS.d/next/Library/2025-01-09-12-06-52.gh-issue-128661.ixx_0z.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 1c86b95e8e5c29..c51ee763890af2 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -45,6 +45,7 @@ import textwrap import typing import weakref +import warnings import types from test.support import captured_stderr, cpython_only, infinite_recursion, requires_docstrings, import_helper, run_code @@ -7273,6 +7274,51 @@ class C(Generic[T]): pass self.assertEqual(get_args(Unpack[tuple[Unpack[Ts]]]), (tuple[Unpack[Ts]],)) +class EvaluateForwardRefTests(BaseTestCase): + def test_evaluate_forward_ref(self): + int_ref = ForwardRef('int') + missing = ForwardRef('missing') + self.assertIs( + typing.evaluate_forward_ref(int_ref, type_params=()), + int, + ) + self.assertIs( + typing.evaluate_forward_ref( + int_ref, type_params=(), format=annotationlib.Format.FORWARDREF, + ), + int, + ) + self.assertIs( + typing.evaluate_forward_ref( + missing, type_params=(), format=annotationlib.Format.FORWARDREF, + ), + missing, + ) + self.assertEqual( + typing.evaluate_forward_ref( + int_ref, type_params=(), format=annotationlib.Format.STRING, + ), + 'int', + ) + + def test_evaluate_forward_ref_no_type_params(self): + ref = ForwardRef('int') + with self.assertWarnsRegex( + DeprecationWarning, + ( + "Failing to pass a value to the 'type_params' parameter " + "of 'typing.evaluate_forward_ref' is deprecated, " + "as it leads to incorrect behaviour" + ), + ): + typing.evaluate_forward_ref(ref) + + # No warnings when `type_params` is passed: + with warnings.catch_warnings(record=True) as w: + typing.evaluate_forward_ref(ref, type_params=()) + self.assertEqual(w, []) + + class CollectionsAbcTests(BaseTestCase): def test_hashable(self): diff --git a/Lib/typing.py b/Lib/typing.py index e69b485422cbd2..66570db7a5bd74 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -1024,7 +1024,7 @@ def evaluate_forward_ref( owner=None, globals=None, locals=None, - type_params=None, + type_params=_sentinel, format=annotationlib.Format.VALUE, _recursive_guard=frozenset(), ): diff --git a/Misc/NEWS.d/next/Library/2025-01-09-12-06-52.gh-issue-128661.ixx_0z.rst b/Misc/NEWS.d/next/Library/2025-01-09-12-06-52.gh-issue-128661.ixx_0z.rst new file mode 100644 index 00000000000000..6c52b3dcc0ed00 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-01-09-12-06-52.gh-issue-128661.ixx_0z.rst @@ -0,0 +1,2 @@ +Fixes :func:`typing.evaluate_forward_ref` not showing deprecation when +``type_params`` arg is not passed.

1 0

gh-128673: Increase coverage of `typing.get_type_hints` (#128674)
by sobolevn Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/43ac9f505903ba806aa6a5d93e6a67beb0… commit: 43ac9f505903ba806aa6a5d93e6a67beb04bebc4 branch: main author: sobolevn <mail(a)sobolevn.me> committer: sobolevn <mail(a)sobolevn.me> date: 2025-01-09T17:25:03+03:00 summary: gh-128673: Increase coverage of `typing.get_type_hints` (#128674) files: M Lib/test/test_typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 45ba7611059e43..1c86b95e8e5c29 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -7152,6 +7152,25 @@ class C: self.assertEqual(get_type_hints(C, format=annotationlib.Format.STRING), {'x': 'undefined'}) + def test_get_type_hints_format_function(self): + def func(x: undefined) -> undefined: ... + + # VALUE + with self.assertRaises(NameError): + get_type_hints(func) + with self.assertRaises(NameError): + get_type_hints(func, format=annotationlib.Format.VALUE) + + # FORWARDREF + self.assertEqual( + get_type_hints(func, format=annotationlib.Format.FORWARDREF), + {'x': ForwardRef('undefined'), 'return': ForwardRef('undefined')}, + ) + + # STRING + self.assertEqual(get_type_hints(func, format=annotationlib.Format.STRING), + {'x': 'undefined', 'return': 'undefined'}) + class GetUtilitiesTestCase(TestCase): def test_get_origin(self):

1 0

gh-118915: Document `PyUnstable_InterpreterState_GetMainModule` (GH-128483)
by encukou Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/ea39c8b08d8f025273bfa5b7a95f7b5984… commit: ea39c8b08d8f025273bfa5b7a95f7b5984dc1e86 branch: main author: Peter Bierma <zintensitydev(a)gmail.com> committer: encukou <encukou(a)gmail.com> date: 2025-01-09T14:54:44+01:00 summary: gh-118915: Document `PyUnstable_InterpreterState_GetMainModule` (GH-128483) files: M Doc/c-api/init.rst diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index dd63dd013e32dc..97996a6f69dd22 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1492,6 +1492,17 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. versionadded:: 3.8 + +.. c:function:: PyObject* PyUnstable_InterpreterState_GetMainModule(PyInterpreterState *interp) + + Return a :term:`strong reference` to the ``__main__`` `module object <moduleobjects>`_ + for the given interpreter. + + The caller must hold the GIL. + + .. versionadded:: 3.13 + + .. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) Type of a frame evaluation function.

1 0

[3.13] gh-124433: fix docs for `asyncio.Queue.task_done` (GH-128669) (#128671)
by kumaraditya303 Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/c57ef49337c6720a76c5f2203d07d6d6d6… commit: c57ef49337c6720a76c5f2203d07d6d6d64f421f branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: kumaraditya303 <kumaraditya(a)python.org> date: 2025-01-09T18:00:55+05:30 summary: [3.13] gh-124433: fix docs for `asyncio.Queue.task_done` (GH-128669) (#128671) gh-124433: fix docs for `asyncio.Queue.task_done` (GH-128669) (cherry picked from commit 4322a318ea98ceeb95d88b7ae6b5cfa3572d2069) Co-authored-by: Kumar Aditya <kumaraditya(a)python.org> files: M Doc/library/asyncio-queue.rst diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 61991bf2f4ed1d..066edd424d150e 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -115,11 +115,11 @@ Queue .. method:: task_done() - Indicate that a formerly enqueued task is complete. + Indicate that a formerly enqueued work item is complete. Used by queue consumers. For each :meth:`~Queue.get` used to - fetch a task, a subsequent call to :meth:`task_done` tells the - queue that the processing on the task is complete. + fetch a work item, a subsequent call to :meth:`task_done` tells the + queue that the processing on the work item is complete. If a :meth:`join` is currently blocking, it will resume when all items have been processed (meaning that a :meth:`task_done`

1 0

[3.12] gh-124433: fix docs for `asyncio.Queue.task_done` (GH-128669) (#128672)
by kumaraditya303 Jan. 9, 2025

Jan. 9, 2025

https://github.com/python/cpython/commit/8bfc68f0f53ab32cf21c2e4d7ad3baca17… commit: 8bfc68f0f53ab32cf21c2e4d7ad3baca177e27fc branch: 3.12 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: kumaraditya303 <kumaraditya(a)python.org> date: 2025-01-09T12:28:15Z summary: [3.12] gh-124433: fix docs for `asyncio.Queue.task_done` (GH-128669) (#128672) gh-124433: fix docs for `asyncio.Queue.task_done` (GH-128669) (cherry picked from commit 4322a318ea98ceeb95d88b7ae6b5cfa3572d2069) Co-authored-by: Kumar Aditya <kumaraditya(a)python.org> files: M Doc/library/asyncio-queue.rst diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 63afc411d9622a..68f386b6a00cde 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -94,11 +94,11 @@ Queue .. method:: task_done() - Indicate that a formerly enqueued task is complete. + Indicate that a formerly enqueued work item is complete. Used by queue consumers. For each :meth:`~Queue.get` used to - fetch a task, a subsequent call to :meth:`task_done` tells the - queue that the processing on the task is complete. + fetch a work item, a subsequent call to :meth:`task_done` tells the + queue that the processing on the work item is complete. If a :meth:`join` is currently blocking, it will resume when all items have been processed (meaning that a :meth:`task_done`

1 0