Mailman 3 January 2025 - Python-checkins

[3.13] gh-128799: Add frame of except* to traceback when wrapping a naked exception (#128971) (#129299)
by iritkatriel Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/a853e2f0760613dc513955ee46143f6d41… commit: a853e2f0760613dc513955ee46143f6d411d352f branch: 3.13 author: Irit Katriel <1055913+iritkatriel(a)users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel(a)users.noreply.github.com> date: 2025-01-30T12:20:13Z summary: [3.13] gh-128799: Add frame of except* to traceback when wrapping a naked exception (#128971) (#129299) files: A Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst M Lib/test/test_traceback.py M Python/ceval.c diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index 2f543bccf566d5..f89fe8b26dc1bf 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -2938,6 +2938,33 @@ def exc(): report = self.get_report(exc) self.assertEqual(report, expected) + def test_exception_group_wrapped_naked(self): + # See gh-128799 + + def exc(): + try: + raise Exception(42) + except* Exception as e: + raise + + expected = (f' + Exception Group Traceback (most recent call last):\n' + f' | File "{__file__}", line {self.callable_line}, in get_exception\n' + f' | exception_or_callable()\n' + f' | ~~~~~~~~~~~~~~~~~~~~~^^\n' + f' | File "{__file__}", line {exc.__code__.co_firstlineno + 3}, in exc\n' + f' | except* Exception as e:\n' + f' | raise\n' + f' | ExceptionGroup: (1 sub-exception)\n' + f' +-+---------------- 1 ----------------\n' + f' | Traceback (most recent call last):\n' + f' | File "{__file__}", line {exc.__code__.co_firstlineno + 2}, in exc\n' + f' | raise Exception(42)\n' + f' | Exception: 42\n' + f' +------------------------------------\n') + + report = self.get_report(exc) + self.assertEqual(report, expected) + def test_KeyboardInterrupt_at_first_line_of_frame(self): # see GH-93249 def f(): diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst b/Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst new file mode 100644 index 00000000000000..eb2361bb5d4525 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst @@ -0,0 +1 @@ +Add frame of ``except*`` to traceback when it wraps a naked exception. diff --git a/Python/ceval.c b/Python/ceval.c index 2f67d40874ba41..763c8688266f9d 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -27,6 +27,7 @@ #include "pycore_setobject.h" // _PySet_Update() #include "pycore_sliceobject.h" // _PyBuildSlice_ConsumeRefs #include "pycore_sysmodule.h" // _PySys_Audit() +#include "pycore_traceback.h" // _PyTraceBack_FromFrame #include "pycore_tuple.h" // _PyTuple_ITEMS() #include "pycore_typeobject.h" // _PySuper_Lookup() #include "pycore_uop_ids.h" // Uops @@ -2018,6 +2019,17 @@ _PyEval_ExceptionGroupMatch(PyObject* exc_value, PyObject *match_type, if (wrapped == NULL) { return -1; } + PyThreadState *tstate = _PyThreadState_GET(); + _PyInterpreterFrame *frame = _PyThreadState_GetFrame(tstate); + PyFrameObject *f = _PyFrame_GetFrameObject(frame); + if (f != NULL) { + PyObject *tb = _PyTraceBack_FromFrame(NULL, f); + if (tb == NULL) { + return -1; + } + PyException_SetTraceback(wrapped, tb); + Py_DECREF(tb); + } *match = wrapped; } *rest = Py_NewRef(Py_None);

1 0

[3.12] gh-128799: Add frame of except* to traceback when wrapping a naked exception (GH-128971) (#129328)
by iritkatriel Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/d2f551db12a779be30e23d031333b7f070… commit: d2f551db12a779be30e23d031333b7f070c9a711 branch: 3.12 author: Irit Katriel <1055913+iritkatriel(a)users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel(a)users.noreply.github.com> date: 2025-01-30T12:02:46Z summary: [3.12] gh-128799: Add frame of except* to traceback when wrapping a naked exception (GH-128971) (#129328) files: A Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst M Lib/test/test_traceback.py M Python/bytecodes.c M Python/ceval.c M Python/generated_cases.c.h diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index 119143e4f3a009..86d509744517ad 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -2157,6 +2157,31 @@ def exc(): report = self.get_report(exc) self.assertEqual(report, expected) + def test_exception_group_wrapped_naked(self): + # See gh-128799 + + def exc(): + try: + raise Exception(42) + except* Exception as e: + raise + + expected = (f' + Exception Group Traceback (most recent call last):\n' + f' | File "{__file__}", line {self.callable_line}, in get_exception\n' + f' | exception_or_callable()\n' + f' | File "{__file__}", line {exc.__code__.co_firstlineno + 3}, in exc\n' + f' | except* Exception as e:\n' + f' | ExceptionGroup: (1 sub-exception)\n' + f' +-+---------------- 1 ----------------\n' + f' | Traceback (most recent call last):\n' + f' | File "{__file__}", line {exc.__code__.co_firstlineno + 2}, in exc\n' + f' | raise Exception(42)\n' + f' | Exception: 42\n' + f' +------------------------------------\n') + + report = self.get_report(exc) + self.assertEqual(report, expected) + def test_KeyboardInterrupt_at_first_line_of_frame(self): # see GH-93249 def f(): diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst b/Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst new file mode 100644 index 00000000000000..eb2361bb5d4525 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2025-01-18-01-06-58.gh-issue-128799.vSNagk.rst @@ -0,0 +1 @@ +Add frame of ``except*`` to traceback when it wraps a naked exception. diff --git a/Python/bytecodes.c b/Python/bytecodes.c index b307edd57dff56..6ee3bda6f703bf 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -2114,7 +2114,7 @@ dummy_func( match = NULL; rest = NULL; - int res = exception_group_match(exc_value, match_type, + int res = exception_group_match(frame, exc_value, match_type, &match, &rest); DECREF_INPUTS(); ERROR_IF(res < 0, error); diff --git a/Python/ceval.c b/Python/ceval.c index b979294b20ff6b..3985b52649cc82 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -20,6 +20,7 @@ #include "pycore_range.h" // _PyRangeIterObject #include "pycore_sliceobject.h" // _PyBuildSlice_ConsumeRefs #include "pycore_sysmodule.h" // _PySys_Audit() +#include "pycore_traceback.h" // _PyTraceBack_FromFrame #include "pycore_tuple.h" // _PyTuple_ITEMS() #include "pycore_typeobject.h" // _PySuper_Lookup() #include "pycore_emscripten_signal.h" // _Py_CHECK_EMSCRIPTEN_SIGNALS @@ -544,6 +545,7 @@ match_class(PyThreadState *tstate, PyObject *subject, PyObject *type, static int do_raise(PyThreadState *tstate, PyObject *exc, PyObject *cause); static int exception_group_match( + _PyInterpreterFrame *frame, PyObject* exc_value, PyObject *match_type, PyObject **match, PyObject **rest); @@ -1856,7 +1858,7 @@ do_raise(PyThreadState *tstate, PyObject *exc, PyObject *cause) */ static int -exception_group_match(PyObject* exc_value, PyObject *match_type, +exception_group_match(_PyInterpreterFrame *frame, PyObject* exc_value, PyObject *match_type, PyObject **match, PyObject **rest) { if (Py_IsNone(exc_value)) { @@ -1883,6 +1885,15 @@ exception_group_match(PyObject* exc_value, PyObject *match_type, if (wrapped == NULL) { return -1; } + PyFrameObject *f = _PyFrame_GetFrameObject(frame); + if (f != NULL) { + PyObject *tb = _PyTraceBack_FromFrame(NULL, f); + if (tb == NULL) { + return -1; + } + PyException_SetTraceback(wrapped, tb); + Py_DECREF(tb); + } *match = wrapped; } *rest = Py_NewRef(Py_None); diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index bbaf589e2ef82c..246d37e771e3c3 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -2952,7 +2952,7 @@ match = NULL; rest = NULL; - int res = exception_group_match(exc_value, match_type, + int res = exception_group_match(frame, exc_value, match_type, &match, &rest); #line 2958 "Python/generated_cases.c.h" Py_DECREF(exc_value);

1 0

clearer error and suggestion when c-analyzer cannot read ignored.tsv (#129423)
by iritkatriel Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/5ab9604683a58e613c6eba309ed89ed209… commit: 5ab9604683a58e613c6eba309ed89ed2092e3e2d branch: main author: Irit Katriel <1055913+iritkatriel(a)users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel(a)users.noreply.github.com> date: 2025-01-30T11:38:52Z summary: clearer error and suggestion when c-analyzer cannot read ignored.tsv (#129423) files: M Tools/c-analyzer/c_analyzer/datafiles.py diff --git a/Tools/c-analyzer/c_analyzer/datafiles.py b/Tools/c-analyzer/c_analyzer/datafiles.py index d5db3bd3ed74ac..79c201a5d3b92c 100644 --- a/Tools/c-analyzer/c_analyzer/datafiles.py +++ b/Tools/c-analyzer/c_analyzer/datafiles.py @@ -104,7 +104,12 @@ def _iter_ignored(infile, relroot): for v in varidinfo) if reason in bogus: reason = None - varid = _info.DeclID.from_row(varidinfo) + try: + varid = _info.DeclID.from_row(varidinfo) + except BaseException as e: + e.add_note(f"Error occurred when processing row {varidinfo} in {infile}.") + e.add_note(f"Could it be that you added a row which is not tab-delimited?") + raise e varid = varid.fix_filename(relroot, formatted=False, fixroot=False) yield varid, reason

1 0

gh-89188: Implement PyUnicode_KIND() as a function (#129412)
by vstinner Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/a810cb89f108f026d70c5b20141b709493… commit: a810cb89f108f026d70c5b20141b709493051d96 branch: main author: Victor Stinner <vstinner(a)python.org> committer: vstinner <vstinner(a)python.org> date: 2025-01-30T11:27:27Z summary: gh-89188: Implement PyUnicode_KIND() as a function (#129412) Implement PyUnicode_KIND() and PyUnicode_DATA() as function, in addition to the macros with the same names. The macros rely on C bit fields which have compiler-specific layout. files: A Misc/NEWS.d/next/C_API/2025-01-29-11-58-38.gh-issue-89188.BsfLr3.rst M Include/cpython/unicodeobject.h M Objects/unicodeobject.c diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 287de52b96202c..cea69dd1280999 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -240,6 +240,8 @@ enum PyUnicode_Kind { PyUnicode_4BYTE_KIND = 4 }; +PyAPI_FUNC(int) PyUnicode_KIND(PyObject *op); + // PyUnicode_KIND(): Return one of the PyUnicode_*_KIND values defined above. // // gh-89653: Converting this macro to a static inline function would introduce @@ -264,13 +266,15 @@ static inline void* _PyUnicode_NONCOMPACT_DATA(PyObject *op) { return data; } -static inline void* PyUnicode_DATA(PyObject *op) { +PyAPI_FUNC(void*) PyUnicode_DATA(PyObject *op); + +static inline void* _PyUnicode_DATA(PyObject *op) { if (PyUnicode_IS_COMPACT(op)) { return _PyUnicode_COMPACT_DATA(op); } return _PyUnicode_NONCOMPACT_DATA(op); } -#define PyUnicode_DATA(op) PyUnicode_DATA(_PyObject_CAST(op)) +#define PyUnicode_DATA(op) _PyUnicode_DATA(_PyObject_CAST(op)) /* Return pointers to the canonical representation cast to unsigned char, Py_UCS2, or Py_UCS4 for direct character access. diff --git a/Misc/NEWS.d/next/C_API/2025-01-29-11-58-38.gh-issue-89188.BsfLr3.rst b/Misc/NEWS.d/next/C_API/2025-01-29-11-58-38.gh-issue-89188.BsfLr3.rst new file mode 100644 index 00000000000000..7ff225a7dc60c7 --- /dev/null +++ b/Misc/NEWS.d/next/C_API/2025-01-29-11-58-38.gh-issue-89188.BsfLr3.rst @@ -0,0 +1,3 @@ +Implement :c:func:`PyUnicode_KIND` and :c:func:`PyUnicode_DATA` as function, +in addition to the macros with the same names. The macros rely on C bit +fields which have compiler-specific layout. Patch by Victor Stinner. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index d9952f764bb178..c6f13f60ad741f 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -16486,3 +16486,24 @@ PyInit__string(void) { return PyModuleDef_Init(&_string_module); } + + +#undef PyUnicode_KIND +int PyUnicode_KIND(PyObject *op) +{ + if (!PyUnicode_Check(op)) { + PyErr_Format(PyExc_TypeError, "expect str, got %T", op); + return -1; + } + return _PyASCIIObject_CAST(op)->state.kind; +} + +#undef PyUnicode_DATA +void* PyUnicode_DATA(PyObject *op) +{ + if (!PyUnicode_Check(op)) { + PyErr_Format(PyExc_TypeError, "expect str, got %T", op); + return NULL; + } + return _PyUnicode_DATA(op); +}

1 0

gh-129005: _pyio.BufferedIO remove copy on readall (#129454)
by vstinner Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/e1c4ba928852eac0b0e0bded1c314e3e36… commit: e1c4ba928852eac0b0e0bded1c314e3e36975286 branch: main author: Cody Maloney <cmaloney(a)users.noreply.github.com> committer: vstinner <vstinner(a)python.org> date: 2025-01-30T11:23:25Z summary: gh-129005: _pyio.BufferedIO remove copy on readall (#129454) Slicing buf and appending chunk would always result in a copy. Commonly in a readall() there is no already read data in buf, and the amount of data read may be large, so the copy is expensive. files: A Misc/NEWS.d/next/Library/2025-01-29-00-00-01.gh-issue-129005.aV_3O8.rst M Lib/_pyio.py diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 76a27910da4d5f..755e0258770891 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -1062,6 +1062,9 @@ def _read_unlocked(self, n=None): if chunk is None: return buf[pos:] or None else: + # Avoid slice + copy if there is no data in buf + if not buf: + return chunk return buf[pos:] + chunk chunks = [buf[pos:]] # Strip the consumed bytes. current_size = 0 diff --git a/Misc/NEWS.d/next/Library/2025-01-29-00-00-01.gh-issue-129005.aV_3O8.rst b/Misc/NEWS.d/next/Library/2025-01-29-00-00-01.gh-issue-129005.aV_3O8.rst new file mode 100644 index 00000000000000..48ee57109be2ff --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-01-29-00-00-01.gh-issue-129005.aV_3O8.rst @@ -0,0 +1,2 @@ +:mod:`!_pyio`: Remove an unnecessary copy when ``_pyio.BufferedReader.read()`` +is called to read all data from a file and has no data already in buffer.

1 0

gh-128911: Add PyImport_ImportModuleAttr() function (#128912)
by vstinner Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/3bebe46d3413195ee18c5c9ada83a35d4f… commit: 3bebe46d3413195ee18c5c9ada83a35d4fd1d0e7 branch: main author: Victor Stinner <vstinner(a)python.org> committer: vstinner <vstinner(a)python.org> date: 2025-01-30T11:17:29Z summary: gh-128911: Add PyImport_ImportModuleAttr() function (#128912) Add PyImport_ImportModuleAttr() and PyImport_ImportModuleAttrString() functions. * Add unit tests. * Replace _PyImport_GetModuleAttr() with PyImport_ImportModuleAttr(). * Replace _PyImport_GetModuleAttrString() with PyImport_ImportModuleAttrString(). * Remove "pycore_import.h" includes, no longer needed. files: A Misc/NEWS.d/next/C_API/2025-01-16-12-47-01.gh-issue-128911.mHVJ4x.rst A Modules/_testcapi/import.c M Doc/c-api/import.rst M Doc/data/refcounts.dat M Doc/whatsnew/3.14.rst M Include/cpython/import.h M Include/internal/pycore_import.h M Lib/test/test_capi/test_import.py M Modules/Setup.stdlib.in M Modules/_ctypes/callbacks.c M Modules/_ctypes/stgdict.c M Modules/_cursesmodule.c M Modules/_datetimemodule.c M Modules/_decimal/_decimal.c M Modules/_elementtree.c M Modules/_json.c M Modules/_lsprof.c M Modules/_operator.c M Modules/_pickle.c M Modules/_sqlite/connection.c M Modules/_sqlite/module.c M Modules/_sre/sre.c M Modules/_testcapi/parts.h M Modules/_testcapimodule.c M Modules/_zoneinfo.c M Modules/arraymodule.c M Modules/cjkcodecs/cjkcodecs.h M Modules/faulthandler.c M Modules/posixmodule.c M Modules/selectmodule.c M Modules/timemodule.c M Objects/abstract.c M Objects/fileobject.c M Objects/memoryobject.c M PCbuild/_testcapi.vcxproj M PCbuild/_testcapi.vcxproj.filters M Parser/pegen.c M Parser/tokenizer/file_tokenizer.c M Python/import.c M Python/pylifecycle.c diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index 6e48644c8fef8b..1cab3ce3061ec9 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -325,3 +325,24 @@ Importing Modules If Python is initialized multiple times, :c:func:`PyImport_AppendInittab` or :c:func:`PyImport_ExtendInittab` must be called before each Python initialization. + + +.. c:function:: PyObject* PyImport_ImportModuleAttr(PyObject *mod_name, PyObject *attr_name) + + Import the module *mod_name* and get its attribute *attr_name*. + + Names must be Python :class:`str` objects. + + Helper function combining :c:func:`PyImport_Import` and + :c:func:`PyObject_GetAttr`. For example, it can raise :exc:`ImportError` if + the module is not found, and :exc:`AttributeError` if the attribute doesn't + exist. + + .. versionadded:: 3.14 + +.. c:function:: PyObject* PyImport_ImportModuleAttrString(const char *mod_name, const char *attr_name) + + Similar to :c:func:`PyImport_ImportModuleAttr`, but names are UTF-8 encoded + strings instead of Python :class:`str` objects. + + .. versionadded:: 3.14 diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index e78754e24e23d8..d709d2d91b0eb0 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -3052,3 +3052,11 @@ _Py_c_quot:Py_complex:divisor:: _Py_c_sum:Py_complex::: _Py_c_sum:Py_complex:left:: _Py_c_sum:Py_complex:right:: + +PyImport_ImportModuleAttr:PyObject*::+1: +PyImport_ImportModuleAttr:PyObject*:mod_name:0: +PyImport_ImportModuleAttr:PyObject*:attr_name:0: + +PyImport_ImportModuleAttrString:PyObject*::+1: +PyImport_ImportModuleAttrString:const char *:mod_name:: +PyImport_ImportModuleAttrString:const char *:attr_name:: diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index 2c10d7fefd44ab..8d4cb94ae2d805 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -1322,6 +1322,11 @@ New features * Add :c:func:`PyUnstable_IsImmortal` for determining whether an object is :term:`immortal`, for debugging purposes. +* Add :c:func:`PyImport_ImportModuleAttr` and + :c:func:`PyImport_ImportModuleAttrString` helper functions to import a module + and get an attribute of the module. + (Contributed by Victor Stinner in :gh:`128911`.) + Limited C API changes --------------------- diff --git a/Include/cpython/import.h b/Include/cpython/import.h index 0fd61c28cafa0e..0ce0b1ee6cce2a 100644 --- a/Include/cpython/import.h +++ b/Include/cpython/import.h @@ -21,3 +21,10 @@ struct _frozen { collection of frozen modules: */ PyAPI_DATA(const struct _frozen *) PyImport_FrozenModules; + +PyAPI_FUNC(PyObject*) PyImport_ImportModuleAttr( + PyObject *mod_name, + PyObject *attr_name); +PyAPI_FUNC(PyObject*) PyImport_ImportModuleAttrString( + const char *mod_name, + const char *attr_name); diff --git a/Include/internal/pycore_import.h b/Include/internal/pycore_import.h index 318c712bdfa174..5fe60df0a92fbc 100644 --- a/Include/internal/pycore_import.h +++ b/Include/internal/pycore_import.h @@ -31,12 +31,6 @@ extern int _PyImport_FixupBuiltin( PyObject *modules ); -// Export for many shared extensions, like '_json' -PyAPI_FUNC(PyObject*) _PyImport_GetModuleAttr(PyObject *, PyObject *); - -// Export for many shared extensions, like '_datetime' -PyAPI_FUNC(PyObject*) _PyImport_GetModuleAttrString(const char *, const char *); - struct _import_runtime_state { /* The builtin modules (defined in config.c). */ diff --git a/Lib/test/test_capi/test_import.py b/Lib/test/test_capi/test_import.py index 94f96728d9174b..25136624ca4ed9 100644 --- a/Lib/test/test_capi/test_import.py +++ b/Lib/test/test_capi/test_import.py @@ -7,6 +7,7 @@ from test.support import import_helper from test.support.warnings_helper import check_warnings +_testcapi = import_helper.import_module('_testcapi') _testlimitedcapi = import_helper.import_module('_testlimitedcapi') NULL = None @@ -148,7 +149,7 @@ def check_frozen_import(self, import_frozen_module): try: self.assertEqual(import_frozen_module('zipimport'), 1) - # import zipimport again + # import zipimport again self.assertEqual(import_frozen_module('zipimport'), 1) finally: sys.modules['zipimport'] = old_zipimport @@ -317,6 +318,59 @@ def test_executecodemoduleobject(self): # CRASHES execute_code_func(NULL, code, NULL, NULL) # CRASHES execute_code_func(name, NULL, NULL, NULL) + def check_importmoduleattr(self, importmoduleattr): + self.assertIs(importmoduleattr('sys', 'argv'), sys.argv) + self.assertIs(importmoduleattr('types', 'ModuleType'), types.ModuleType) + + # module name containing a dot + attr = importmoduleattr('email.message', 'Message') + from email.message import Message + self.assertIs(attr, Message) + + with self.assertRaises(ImportError): + # nonexistent module + importmoduleattr('nonexistentmodule', 'attr') + with self.assertRaises(AttributeError): + # nonexistent attribute + importmoduleattr('sys', 'nonexistentattr') + with self.assertRaises(AttributeError): + # attribute name containing a dot + importmoduleattr('sys', 'implementation.name') + + def test_importmoduleattr(self): + # Test PyImport_ImportModuleAttr() + importmoduleattr = _testcapi.PyImport_ImportModuleAttr + self.check_importmoduleattr(importmoduleattr) + + # Invalid module name type + for mod_name in (object(), 123, b'bytes'): + with self.subTest(mod_name=mod_name): + with self.assertRaises(TypeError): + importmoduleattr(mod_name, "attr") + + # Invalid attribute name type + for attr_name in (object(), 123, b'bytes'): + with self.subTest(attr_name=attr_name): + with self.assertRaises(TypeError): + importmoduleattr("sys", attr_name) + + with self.assertRaises(SystemError): + importmoduleattr(NULL, "argv") + # CRASHES importmoduleattr("sys", NULL) + + def test_importmoduleattrstring(self): + # Test PyImport_ImportModuleAttrString() + importmoduleattr = _testcapi.PyImport_ImportModuleAttrString + self.check_importmoduleattr(importmoduleattr) + + with self.assertRaises(UnicodeDecodeError): + importmoduleattr(b"sys\xff", "argv") + with self.assertRaises(UnicodeDecodeError): + importmoduleattr("sys", b"argv\xff") + + # CRASHES importmoduleattr(NULL, "argv") + # CRASHES importmoduleattr("sys", NULL) + # TODO: test PyImport_GetImporter() # TODO: test PyImport_ReloadModule() # TODO: test PyImport_ExtendInittab() diff --git a/Misc/NEWS.d/next/C_API/2025-01-16-12-47-01.gh-issue-128911.mHVJ4x.rst b/Misc/NEWS.d/next/C_API/2025-01-16-12-47-01.gh-issue-128911.mHVJ4x.rst new file mode 100644 index 00000000000000..d32cd00cd5d605 --- /dev/null +++ b/Misc/NEWS.d/next/C_API/2025-01-16-12-47-01.gh-issue-128911.mHVJ4x.rst @@ -0,0 +1,3 @@ +Add :c:func:`PyImport_ImportModuleAttr` and :c:func:`PyImport_ImportModuleAttrString` +helper functions to import a module and get an attribute of the module. Patch +by Victor Stinner. diff --git a/Modules/Setup.stdlib.in b/Modules/Setup.stdlib.in index 6b6a8ae57a5119..563bbc1bda6223 100644 --- a/Modules/Setup.stdlib.in +++ b/Modules/Setup.stdlib.in @@ -162,7 +162,7 @@ @MODULE__XXTESTFUZZ_TRUE@_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c @MODULE__TESTBUFFER_TRUE@_testbuffer _testbuffer.c @MODULE__TESTINTERNALCAPI_TRUE@_testinternalcapi _testinternalcapi.c _testinternalcapi/test_lock.c _testinternalcapi/pytime.c _testinternalcapi/set.c _testinternalcapi/test_critical_sections.c -@MODULE__TESTCAPI_TRUE@_testcapi _testcapimodule.c _testcapi/vectorcall.c _testcapi/heaptype.c _testcapi/abstract.c _testcapi/unicode.c _testcapi/dict.c _testcapi/set.c _testcapi/list.c _testcapi/tuple.c _testcapi/getargs.c _testcapi/datetime.c _testcapi/docstring.c _testcapi/mem.c _testcapi/watchers.c _testcapi/long.c _testcapi/float.c _testcapi/complex.c _testcapi/numbers.c _testcapi/structmember.c _testcapi/exceptions.c _testcapi/code.c _testcapi/buffer.c _testcapi/pyatomic.c _testcapi/run.c _testcapi/file.c _testcapi/codec.c _testcapi/immortal.c _testcapi/gc.c _testcapi/hash.c _testcapi/time.c _testcapi/bytes.c _testcapi/object.c _testcapi/monitoring.c _testcapi/config.c +@MODULE__TESTCAPI_TRUE@_testcapi _testcapimodule.c _testcapi/vectorcall.c _testcapi/heaptype.c _testcapi/abstract.c _testcapi/unicode.c _testcapi/dict.c _testcapi/set.c _testcapi/list.c _testcapi/tuple.c _testcapi/getargs.c _testcapi/datetime.c _testcapi/docstring.c _testcapi/mem.c _testcapi/watchers.c _testcapi/long.c _testcapi/float.c _testcapi/complex.c _testcapi/numbers.c _testcapi/structmember.c _testcapi/exceptions.c _testcapi/code.c _testcapi/buffer.c _testcapi/pyatomic.c _testcapi/run.c _testcapi/file.c _testcapi/codec.c _testcapi/immortal.c _testcapi/gc.c _testcapi/hash.c _testcapi/time.c _testcapi/bytes.c _testcapi/object.c _testcapi/monitoring.c _testcapi/config.c _testcapi/import.c @MODULE__TESTLIMITEDCAPI_TRUE@_testlimitedcapi _testlimitedcapi.c _testlimitedcapi/abstract.c _testlimitedcapi/bytearray.c _testlimitedcapi/bytes.c _testlimitedcapi/codec.c _testlimitedcapi/complex.c _testlimitedcapi/dict.c _testlimitedcapi/eval.c _testlimitedcapi/float.c _testlimitedcapi/heaptype_relative.c _testlimitedcapi/import.c _testlimitedcapi/list.c _testlimitedcapi/long.c _testlimitedcapi/object.c _testlimitedcapi/pyos.c _testlimitedcapi/set.c _testlimitedcapi/sys.c _testlimitedcapi/tuple.c _testlimitedcapi/unicode.c _testlimitedcapi/vectorcall_limited.c _testlimitedcapi/version.c @MODULE__TESTCLINIC_TRUE@_testclinic _testclinic.c @MODULE__TESTCLINIC_LIMITED_TRUE@_testclinic_limited _testclinic_limited.c diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index 89c0749a093765..591206a78035c5 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -492,7 +492,7 @@ long Call_GetClassObject(REFCLSID rclsid, REFIID riid, LPVOID *ppv) if (context == NULL) context = PyUnicode_InternFromString("_ctypes.DllGetClassObject"); - func = _PyImport_GetModuleAttrString("ctypes", "DllGetClassObject"); + func = PyImport_ImportModuleAttrString("ctypes", "DllGetClassObject"); if (!func) { PyErr_WriteUnraisable(context ? context : Py_None); /* There has been a warning before about this already */ diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c index 5ca5b62427600d..d63a46a3bc23d2 100644 --- a/Modules/_ctypes/stgdict.c +++ b/Modules/_ctypes/stgdict.c @@ -257,7 +257,7 @@ PyCStructUnionType_update_stginfo(PyObject *type, PyObject *fields, int isStruct goto error; } - PyObject *layout_func = _PyImport_GetModuleAttrString("ctypes._layout", + PyObject *layout_func = PyImport_ImportModuleAttrString("ctypes._layout", "get_layout"); if (!layout_func) { goto error; diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index c6835738348ff9..7213a5be07de4b 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -226,7 +226,7 @@ _PyCursesCheckFunction(int called, const char *funcname) if (called == TRUE) { return 1; } - PyObject *exc = _PyImport_GetModuleAttrString("_curses", "error"); + PyObject *exc = PyImport_ImportModuleAttrString("_curses", "error"); if (exc != NULL) { PyErr_Format(exc, "must call %s() first", funcname); Py_DECREF(exc); diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c index ff2e6d6a098ad9..a486af7833c2c6 100644 --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -1839,7 +1839,7 @@ wrap_strftime(PyObject *object, PyObject *format, PyObject *timetuple, assert(object && format && timetuple); assert(PyUnicode_Check(format)); - PyObject *strftime = _PyImport_GetModuleAttrString("time", "strftime"); + PyObject *strftime = PyImport_ImportModuleAttrString("time", "strftime"); if (strftime == NULL) { return NULL; } @@ -2022,7 +2022,7 @@ static PyObject * time_time(void) { PyObject *result = NULL; - PyObject *time = _PyImport_GetModuleAttrString("time", "time"); + PyObject *time = PyImport_ImportModuleAttrString("time", "time"); if (time != NULL) { result = PyObject_CallNoArgs(time); @@ -2040,7 +2040,7 @@ build_struct_time(int y, int m, int d, int hh, int mm, int ss, int dstflag) PyObject *struct_time; PyObject *result; - struct_time = _PyImport_GetModuleAttrString("time", "struct_time"); + struct_time = PyImport_ImportModuleAttrString("time", "struct_time"); if (struct_time == NULL) { return NULL; } diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index 78cf6b1426493b..3dcb3e9870c8a4 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -3474,7 +3474,7 @@ pydec_format(PyObject *dec, PyObject *context, PyObject *fmt, decimal_state *sta PyObject *u; if (state->PyDecimal == NULL) { - state->PyDecimal = _PyImport_GetModuleAttrString("_pydecimal", "Decimal"); + state->PyDecimal = PyImport_ImportModuleAttrString("_pydecimal", "Decimal"); if (state->PyDecimal == NULL) { return NULL; } diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c index 355f322d304c2f..b5b0b82571f882 100644 --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -16,7 +16,6 @@ #endif #include "Python.h" -#include "pycore_import.h" // _PyImport_GetModuleAttrString() #include "pycore_pyhash.h" // _Py_HashSecret #include <stddef.h> // offsetof() @@ -4393,7 +4392,7 @@ module_exec(PyObject *m) CREATE_TYPE(m, st->Element_Type, &element_spec); CREATE_TYPE(m, st->XMLParser_Type, &xmlparser_spec); - st->deepcopy_obj = _PyImport_GetModuleAttrString("copy", "deepcopy"); + st->deepcopy_obj = PyImport_ImportModuleAttrString("copy", "deepcopy"); if (st->deepcopy_obj == NULL) { goto error; } @@ -4403,7 +4402,7 @@ module_exec(PyObject *m) goto error; /* link against pyexpat */ - if (!(st->expat_capsule = _PyImport_GetModuleAttrString("pyexpat", "expat_CAPI"))) + if (!(st->expat_capsule = PyImport_ImportModuleAttrString("pyexpat", "expat_CAPI"))) goto error; if (!(st->expat_capi = PyCapsule_GetPointer(st->expat_capsule, PyExpat_CAPSULE_NAME))) goto error; diff --git a/Modules/_json.c b/Modules/_json.c index 31a5e935e13ad9..5532e252819bbd 100644 --- a/Modules/_json.c +++ b/Modules/_json.c @@ -302,7 +302,7 @@ raise_errmsg(const char *msg, PyObject *s, Py_ssize_t end) /* Use JSONDecodeError exception to raise a nice looking ValueError subclass */ _Py_DECLARE_STR(json_decoder, "json.decoder"); PyObject *JSONDecodeError = - _PyImport_GetModuleAttr(&_Py_STR(json_decoder), &_Py_ID(JSONDecodeError)); + PyImport_ImportModuleAttr(&_Py_STR(json_decoder), &_Py_ID(JSONDecodeError)); if (JSONDecodeError == NULL) { return; } diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c index 51ad9fc7da8492..29d9e5b6ef2cbe 100644 --- a/Modules/_lsprof.c +++ b/Modules/_lsprof.c @@ -775,7 +775,7 @@ _lsprof_Profiler_enable_impl(ProfilerObject *self, int subcalls, return NULL; } - PyObject* monitoring = _PyImport_GetModuleAttrString("sys", "monitoring"); + PyObject* monitoring = PyImport_ImportModuleAttrString("sys", "monitoring"); if (!monitoring) { return NULL; } @@ -857,7 +857,7 @@ _lsprof_Profiler_disable_impl(ProfilerObject *self) } if (self->flags & POF_ENABLED) { PyObject* result = NULL; - PyObject* monitoring = _PyImport_GetModuleAttrString("sys", "monitoring"); + PyObject* monitoring = PyImport_ImportModuleAttrString("sys", "monitoring"); if (!monitoring) { return NULL; @@ -973,7 +973,7 @@ profiler_init_impl(ProfilerObject *self, PyObject *timer, double timeunit, Py_XSETREF(self->externalTimer, Py_XNewRef(timer)); self->tool_id = PY_MONITORING_PROFILER_ID; - PyObject* monitoring = _PyImport_GetModuleAttrString("sys", "monitoring"); + PyObject* monitoring = PyImport_ImportModuleAttrString("sys", "monitoring"); if (!monitoring) { return -1; } diff --git a/Modules/_operator.c b/Modules/_operator.c index ce3ef015710223..59987b8f143da2 100644 --- a/Modules/_operator.c +++ b/Modules/_operator.c @@ -1868,7 +1868,7 @@ methodcaller_reduce(methodcallerobject *mc, PyObject *Py_UNUSED(ignored)) PyObject *constructor; PyObject *newargs[2]; - partial = _PyImport_GetModuleAttrString("functools", "partial"); + partial = PyImport_ImportModuleAttrString("functools", "partial"); if (!partial) return NULL; diff --git a/Modules/_pickle.c b/Modules/_pickle.c index a6cfb2deeb23c1..5641f93391c551 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -362,7 +362,7 @@ _Pickle_InitState(PickleState *st) } Py_CLEAR(compat_pickle); - st->codecs_encode = _PyImport_GetModuleAttrString("codecs", "encode"); + st->codecs_encode = PyImport_ImportModuleAttrString("codecs", "encode"); if (st->codecs_encode == NULL) { goto error; } @@ -373,7 +373,7 @@ _Pickle_InitState(PickleState *st) goto error; } - st->partial = _PyImport_GetModuleAttrString("functools", "partial"); + st->partial = PyImport_ImportModuleAttrString("functools", "partial"); if (!st->partial) goto error; diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 62598ecc864120..0c98f5065ee303 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -34,7 +34,6 @@ #include "prepare_protocol.h" #include "util.h" -#include "pycore_import.h" // _PyImport_GetModuleAttrString() #include "pycore_modsupport.h" // _PyArg_NoKeywords() #include "pycore_pyerrors.h" // _PyErr_ChainExceptions1() #include "pycore_pylifecycle.h" // _Py_IsInterpreterFinalizing() @@ -2000,7 +1999,7 @@ pysqlite_connection_iterdump_impl(pysqlite_Connection *self, return NULL; } - PyObject *iterdump = _PyImport_GetModuleAttrString(MODULE_NAME ".dump", "_iterdump"); + PyObject *iterdump = PyImport_ImportModuleAttrString(MODULE_NAME ".dump", "_iterdump"); if (!iterdump) { if (!PyErr_Occurred()) { PyErr_SetString(self->OperationalError, diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index 698e81d9b897d0..73d55fb44e2e15 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -33,8 +33,6 @@ #include "row.h" #include "blob.h" -#include "pycore_import.h" // _PyImport_GetModuleAttrString() - #if SQLITE_VERSION_NUMBER < 3015002 #error "SQLite 3.15.2 or higher required" #endif @@ -234,7 +232,7 @@ static int load_functools_lru_cache(PyObject *module) { pysqlite_state *state = pysqlite_get_state(module); - state->lru_cache = _PyImport_GetModuleAttrString("functools", "lru_cache"); + state->lru_cache = PyImport_ImportModuleAttrString("functools", "lru_cache"); if (state->lru_cache == NULL) { return -1; } diff --git a/Modules/_sre/sre.c b/Modules/_sre/sre.c index d0025dd21e045b..0d8d4843d33c1b 100644 --- a/Modules/_sre/sre.c +++ b/Modules/_sre/sre.c @@ -1169,7 +1169,7 @@ compile_template(_sremodulestate *module_state, /* delegate to Python code */ PyObject *func = module_state->compile_template; if (func == NULL) { - func = _PyImport_GetModuleAttrString("re", "_compile_template"); + func = PyImport_ImportModuleAttrString("re", "_compile_template"); if (func == NULL) { return NULL; } diff --git a/Modules/_testcapi/import.c b/Modules/_testcapi/import.c new file mode 100644 index 00000000000000..27d37498f3cd83 --- /dev/null +++ b/Modules/_testcapi/import.c @@ -0,0 +1,44 @@ +#include "parts.h" +#include "util.h" + +// Test PyImport_ImportModuleAttr() +static PyObject * +pyimport_importmoduleattr(PyObject *self, PyObject *args) +{ + PyObject *mod_name, *attr_name; + if (!PyArg_ParseTuple(args, "OO", &mod_name, &attr_name)) { + return NULL; + } + NULLABLE(mod_name); + NULLABLE(attr_name); + + return PyImport_ImportModuleAttr(mod_name, attr_name); +} + + +// Test PyImport_ImportModuleAttrString() +static PyObject * +pyimport_importmoduleattrstring(PyObject *self, PyObject *args) +{ + const char *mod_name, *attr_name; + Py_ssize_t len; + if (!PyArg_ParseTuple(args, "z#z#", &mod_name, &len, &attr_name, &len)) { + return NULL; + } + + return PyImport_ImportModuleAttrString(mod_name, attr_name); +} + + +static PyMethodDef test_methods[] = { + {"PyImport_ImportModuleAttr", pyimport_importmoduleattr, METH_VARARGS}, + {"PyImport_ImportModuleAttrString", pyimport_importmoduleattrstring, METH_VARARGS}, + {NULL}, +}; + +int +_PyTestCapi_Init_Import(PyObject *m) +{ + return PyModule_AddFunctions(m, test_methods); +} + diff --git a/Modules/_testcapi/parts.h b/Modules/_testcapi/parts.h index 65ba77596c760e..792552d8097312 100644 --- a/Modules/_testcapi/parts.h +++ b/Modules/_testcapi/parts.h @@ -61,5 +61,6 @@ int _PyTestCapi_Init_Time(PyObject *module); int _PyTestCapi_Init_Monitoring(PyObject *module); int _PyTestCapi_Init_Object(PyObject *module); int _PyTestCapi_Init_Config(PyObject *mod); +int _PyTestCapi_Init_Import(PyObject *mod); #endif // Py_TESTCAPI_PARTS_H diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index c405a352ed74a1..4da23ba82d5756 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -4401,6 +4401,9 @@ PyInit__testcapi(void) if (_PyTestCapi_Init_Config(m) < 0) { return NULL; } + if (_PyTestCapi_Init_Import(m) < 0) { + return NULL; + } PyState_AddModule(m, &_testcapimodule); return m; diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index c5292575c22f23..1fcea9ce8b1261 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -782,7 +782,7 @@ zoneinfo_reduce(PyObject *obj_self, PyObject *unused) if (self->source == SOURCE_FILE) { // Objects constructed from files cannot be pickled. PyObject *pickle_error = - _PyImport_GetModuleAttrString("pickle", "PicklingError"); + PyImport_ImportModuleAttrString("pickle", "PicklingError"); if (pickle_error == NULL) { return NULL; } @@ -2554,7 +2554,7 @@ static PyObject * new_weak_cache(void) { PyObject *WeakValueDictionary = - _PyImport_GetModuleAttrString("weakref", "WeakValueDictionary"); + PyImport_ImportModuleAttrString("weakref", "WeakValueDictionary"); if (WeakValueDictionary == NULL) { return NULL; } @@ -2732,12 +2732,12 @@ zoneinfomodule_exec(PyObject *m) /* Populate imports */ state->_tzpath_find_tzfile = - _PyImport_GetModuleAttrString("zoneinfo._tzpath", "find_tzfile"); + PyImport_ImportModuleAttrString("zoneinfo._tzpath", "find_tzfile"); if (state->_tzpath_find_tzfile == NULL) { goto error; } - state->io_open = _PyImport_GetModuleAttrString("io", "open"); + state->io_open = PyImport_ImportModuleAttrString("io", "open"); if (state->io_open == NULL) { goto error; } diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c index 28c6a0ae05c598..dc1729a7a3a558 100644 --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2285,7 +2285,7 @@ array_array___reduce_ex___impl(arrayobject *self, PyTypeObject *cls, assert(state != NULL); if (state->array_reconstructor == NULL) { - state->array_reconstructor = _PyImport_GetModuleAttrString( + state->array_reconstructor = PyImport_ImportModuleAttrString( "array", "_array_reconstructor"); if (state->array_reconstructor == NULL) { return NULL; @@ -3206,7 +3206,7 @@ array_modexec(PyObject *m) return -1; } - PyObject *mutablesequence = _PyImport_GetModuleAttrString( + PyObject *mutablesequence = PyImport_ImportModuleAttrString( "collections.abc", "MutableSequence"); if (!mutablesequence) { Py_DECREF((PyObject *)state->ArrayType); diff --git a/Modules/cjkcodecs/cjkcodecs.h b/Modules/cjkcodecs/cjkcodecs.h index 2b446ba5226ac0..737a7a042753a9 100644 --- a/Modules/cjkcodecs/cjkcodecs.h +++ b/Modules/cjkcodecs/cjkcodecs.h @@ -13,7 +13,6 @@ #include "Python.h" #include "multibytecodec.h" -#include "pycore_import.h" // _PyImport_GetModuleAttrString() /* a unicode "undefined" code point */ @@ -299,7 +298,7 @@ add_codecs(cjkcodecs_module_state *st) \ static PyObject * getmultibytecodec(void) { - return _PyImport_GetModuleAttrString("_multibytecodec", "__create_codec"); + return PyImport_ImportModuleAttrString("_multibytecodec", "__create_codec"); } static void diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c index b44b964b29484b..a15ced22677ab7 100644 --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -1346,7 +1346,7 @@ PyInit_faulthandler(void) static int faulthandler_init_enable(void) { - PyObject *enable = _PyImport_GetModuleAttrString("faulthandler", "enable"); + PyObject *enable = PyImport_ImportModuleAttrString("faulthandler", "enable"); if (enable == NULL) { return -1; } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index a35a848a7ca4b8..6d9b365ea9ceb2 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -9877,7 +9877,7 @@ wait_helper(PyObject *module, pid_t pid, int status, struct rusage *ru) memset(ru, 0, sizeof(*ru)); } - struct_rusage = _PyImport_GetModuleAttrString("resource", "struct_rusage"); + struct_rusage = PyImport_ImportModuleAttrString("resource", "struct_rusage"); if (struct_rusage == NULL) return NULL; diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c index e14e114a6dafd0..c75e2ba28c5b4e 100644 --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -14,7 +14,6 @@ #include "Python.h" #include "pycore_fileutils.h" // _Py_set_inheritable() -#include "pycore_import.h" // _PyImport_GetModuleAttrString() #include "pycore_time.h" // _PyTime_FromSecondsObject() #include <stdbool.h> @@ -1996,7 +1995,7 @@ kqueue_tracking_init(PyObject *module) { // Register a callback to invalidate kqueues with open fds after fork. PyObject *register_at_fork = NULL, *cb = NULL, *args = NULL, *kwargs = NULL, *result = NULL; - register_at_fork = _PyImport_GetModuleAttrString("posix", + register_at_fork = PyImport_ImportModuleAttrString("posix", "register_at_fork"); if (register_at_fork == NULL) { goto finally; diff --git a/Modules/timemodule.c b/Modules/timemodule.c index 5d0cd52a93a2d3..8d2cbff662b9a3 100644 --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -979,7 +979,7 @@ time_strptime(PyObject *self, PyObject *args) { PyObject *func, *result; - func = _PyImport_GetModuleAttrString("_strptime", "_strptime_time"); + func = PyImport_ImportModuleAttrString("_strptime", "_strptime_time"); if (!func) { return NULL; } diff --git a/Objects/abstract.c b/Objects/abstract.c index c92ef10aa79648..db7b9263711f68 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -583,7 +583,7 @@ PyBuffer_SizeFromFormat(const char *format) PyObject *fmt = NULL; Py_ssize_t itemsize = -1; - calcsize = _PyImport_GetModuleAttrString("struct", "calcsize"); + calcsize = PyImport_ImportModuleAttrString("struct", "calcsize"); if (calcsize == NULL) { goto done; } diff --git a/Objects/fileobject.c b/Objects/fileobject.c index c377d1bb28b56f..7025b5bcffc1c8 100644 --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -34,7 +34,7 @@ PyFile_FromFd(int fd, const char *name, const char *mode, int buffering, const c PyObject *open, *stream; /* import _io in case we are being used to open io.py */ - open = _PyImport_GetModuleAttrString("_io", "open"); + open = PyImport_ImportModuleAttrString("_io", "open"); if (open == NULL) return NULL; stream = PyObject_CallFunction(open, "isisssO", fd, mode, @@ -506,7 +506,7 @@ PyFile_OpenCodeObject(PyObject *path) if (hook) { f = hook(path, _PyRuntime.open_code_userdata); } else { - PyObject *open = _PyImport_GetModuleAttrString("_io", "open"); + PyObject *open = PyImport_ImportModuleAttrString("_io", "open"); if (open) { f = PyObject_CallFunction(open, "Os", path, "rb"); Py_DECREF(open); diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c index ea4d24dc690768..331363b2babbef 100644 --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -2083,7 +2083,7 @@ struct_get_unpacker(const char *fmt, Py_ssize_t itemsize) PyObject *format = NULL; struct unpacker *x = NULL; - Struct = _PyImport_GetModuleAttrString("struct", "Struct"); + Struct = PyImport_ImportModuleAttrString("struct", "Struct"); if (Struct == NULL) return NULL; diff --git a/PCbuild/_testcapi.vcxproj b/PCbuild/_testcapi.vcxproj index c41235eac356af..733bb69acb16e2 100644 --- a/PCbuild/_testcapi.vcxproj +++ b/PCbuild/_testcapi.vcxproj @@ -127,6 +127,7 @@ <ClCompile Include="..\Modules\_testcapi\run.c" /> <ClCompile Include="..\Modules\_testcapi\monitoring.c" /> <ClCompile Include="..\Modules\_testcapi\config.c" /> + <ClCompile Include="..\Modules\_testcapi\import.c" /> </ItemGroup> <ItemGroup> <ResourceCompile Include="..\PC\python_nt.rc" /> diff --git a/PCbuild/_testcapi.vcxproj.filters b/PCbuild/_testcapi.vcxproj.filters index 0a00df655deefc..e8ddd537674a9c 100644 --- a/PCbuild/_testcapi.vcxproj.filters +++ b/PCbuild/_testcapi.vcxproj.filters @@ -114,6 +114,9 @@ <ClCompile Include="..\Modules\_testcapi\config.c"> <Filter>Source Files</Filter> </ClCompile> + <ClCompile Include="..\Modules\_testcapi\import.c"> + <Filter>Source Files</Filter> + </ClCompile> </ItemGroup> <ItemGroup> <ResourceCompile Include="..\PC\python_nt.rc"> diff --git a/Parser/pegen.c b/Parser/pegen.c index bb98e7b184a4dc..83b0022e47d619 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -111,7 +111,7 @@ init_normalization(Parser *p) if (p->normalize) { return 1; } - p->normalize = _PyImport_GetModuleAttrString("unicodedata", "normalize"); + p->normalize = PyImport_ImportModuleAttrString("unicodedata", "normalize"); if (!p->normalize) { return 0; diff --git a/Parser/tokenizer/file_tokenizer.c b/Parser/tokenizer/file_tokenizer.c index 2750527da484aa..efe9fb9b56abaf 100644 --- a/Parser/tokenizer/file_tokenizer.c +++ b/Parser/tokenizer/file_tokenizer.c @@ -158,7 +158,7 @@ fp_setreadl(struct tok_state *tok, const char* enc) return 0; } - open = _PyImport_GetModuleAttrString("io", "open"); + open = PyImport_ImportModuleAttrString("io", "open"); if (open == NULL) { return 0; } diff --git a/Python/import.c b/Python/import.c index b3648e24d0e064..dd7a0b4b1ed8de 100644 --- a/Python/import.c +++ b/Python/import.c @@ -4111,7 +4111,7 @@ init_zipimport(PyThreadState *tstate, int verbose) PySys_WriteStderr("# installing zipimport hook\n"); } - PyObject *zipimporter = _PyImport_GetModuleAttrString("zipimport", "zipimporter"); + PyObject *zipimporter = PyImport_ImportModuleAttrString("zipimport", "zipimporter"); if (zipimporter == NULL) { _PyErr_Clear(tstate); /* No zipimporter object -- okay */ if (verbose) { @@ -4174,7 +4174,7 @@ _PyImport_FiniExternal(PyInterpreterState *interp) /******************/ PyObject * -_PyImport_GetModuleAttr(PyObject *modname, PyObject *attrname) +PyImport_ImportModuleAttr(PyObject *modname, PyObject *attrname) { PyObject *mod = PyImport_Import(modname); if (mod == NULL) { @@ -4186,7 +4186,7 @@ _PyImport_GetModuleAttr(PyObject *modname, PyObject *attrname) } PyObject * -_PyImport_GetModuleAttrString(const char *modname, const char *attrname) +PyImport_ImportModuleAttrString(const char *modname, const char *attrname) { PyObject *pmodname = PyUnicode_FromString(modname); if (pmodname == NULL) { @@ -4197,7 +4197,7 @@ _PyImport_GetModuleAttrString(const char *modname, const char *attrname) Py_DECREF(pmodname); return NULL; } - PyObject *result = _PyImport_GetModuleAttr(pmodname, pattrname); + PyObject *result = PyImport_ImportModuleAttr(pmodname, pattrname); Py_DECREF(pattrname); Py_DECREF(pmodname); return result; diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 00a98af998cfce..7031d743174650 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -2609,7 +2609,7 @@ create_stdio(const PyConfig *config, PyObject* io, #ifdef HAVE_WINDOWS_CONSOLE_IO /* Windows console IO is always UTF-8 encoded */ - PyTypeObject *winconsoleio_type = (PyTypeObject *)_PyImport_GetModuleAttr( + PyTypeObject *winconsoleio_type = (PyTypeObject *)PyImport_ImportModuleAttr( &_Py_ID(_io), &_Py_ID(_WindowsConsoleIO)); if (winconsoleio_type == NULL) { goto error; @@ -2714,7 +2714,7 @@ init_set_builtins_open(void) goto error; } - if (!(wrapper = _PyImport_GetModuleAttrString("io", "open"))) { + if (!(wrapper = PyImport_ImportModuleAttrString("io", "open"))) { goto error; }

1 0

gh-129005: Align FileIO.readall() allocation (#129458)
by vstinner Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/f927204f64b3f8dbecec784e05bc8e25d2… commit: f927204f64b3f8dbecec784e05bc8e25d2a78b2e branch: main author: Cody Maloney <cmaloney(a)users.noreply.github.com> committer: vstinner <vstinner(a)python.org> date: 2025-01-30T11:14:23Z summary: gh-129005: Align FileIO.readall() allocation (#129458) Both now use a pre-allocated buffer of length `bufsize`, fill it using a readinto(), and have matching "expand buffer" logic. On my machine this takes: `./python -m test -M8g -uall test_largefile -m test_large_read -v` from ~3.7 seconds to ~3.4 seconds. files: A Misc/NEWS.d/next/Library/2025-01-28-21-22-44.gh-issue-129005.h57i9j.rst M Lib/_pyio.py diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 023478aa78c6a0..76a27910da4d5f 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -1674,22 +1674,31 @@ def readall(self): except OSError: pass - result = bytearray() + result = bytearray(bufsize) + bytes_read = 0 while True: - if len(result) >= bufsize: - bufsize = len(result) - bufsize += max(bufsize, DEFAULT_BUFFER_SIZE) - n = bufsize - len(result) + if bytes_read >= bufsize: + # Parallels _io/fileio.c new_buffersize + if bufsize > 65536: + addend = bufsize >> 3 + else: + addend = bufsize + 256 + if addend < DEFAULT_BUFFER_SIZE: + addend = DEFAULT_BUFFER_SIZE + bufsize += addend + result[bytes_read:bufsize] = b'\0' + assert bufsize - bytes_read > 0, "Should always try and read at least one byte" try: - chunk = os.read(self._fd, n) + n = os.readinto(self._fd, memoryview(result)[bytes_read:]) except BlockingIOError: - if result: + if bytes_read > 0: break return None - if not chunk: # reached the end of the file + if n == 0: # reached the end of the file break - result += chunk + bytes_read += n + del result[bytes_read:] return bytes(result) def readinto(self, buffer): diff --git a/Misc/NEWS.d/next/Library/2025-01-28-21-22-44.gh-issue-129005.h57i9j.rst b/Misc/NEWS.d/next/Library/2025-01-28-21-22-44.gh-issue-129005.h57i9j.rst new file mode 100644 index 00000000000000..c76fb05e196f87 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-01-28-21-22-44.gh-issue-129005.h57i9j.rst @@ -0,0 +1,2 @@ +``_pyio.FileIO.readall()`` now allocates, resizes, and fills a data buffer using +the same algorithm ``_io.FileIO.readall()`` uses.

1 0

gh-129467: Fix compiler warning in mpdecimal word_to_string() (#116346)
by vstinner Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/6c63afc3be9dd612f587fe7869c1f317d5… commit: 6c63afc3be9dd612f587fe7869c1f317d5ea37cc branch: main author: Sergey B Kirpichev <skirpichev(a)gmail.com> committer: vstinner <vstinner(a)python.org> date: 2025-01-30T11:57:55+01:00 summary: gh-129467: Fix compiler warning in mpdecimal word_to_string() (#116346) Turn off false-positive -Wstringop-overflow in word_to_string(). files: M Misc/sbom.spdx.json M Modules/_decimal/libmpdec/io.c diff --git a/Misc/sbom.spdx.json b/Misc/sbom.spdx.json index b4d785f65639a5..316c266b7e4fd6 100644 --- a/Misc/sbom.spdx.json +++ b/Misc/sbom.spdx.json @@ -1280,11 +1280,11 @@ "checksums": [ { "algorithm": "SHA1", - "checksumValue": "9dcb50e3f9c3245972731be5da0b28e7583198d9" + "checksumValue": "5d6fdd98730584f74f7b731da6e488fe234504b3" }, { "algorithm": "SHA256", - "checksumValue": "7cac49fef5e9d952ec9390bf81c54d83f1b5da32fdf76091c2f0770ed943b7fe" + "checksumValue": "d74f365463166891f62e1326d22b2d39d865776b7ea5e0df2aea5eede4d85b0f" } ], "fileName": "Modules/_decimal/libmpdec/io.c" diff --git a/Modules/_decimal/libmpdec/io.c b/Modules/_decimal/libmpdec/io.c index 4e95b8964c8e5d..bdcca001659bc0 100644 --- a/Modules/_decimal/libmpdec/io.c +++ b/Modules/_decimal/libmpdec/io.c @@ -347,6 +347,10 @@ mpd_qset_string_exact(mpd_t *dec, const char *s, uint32_t *status) or the location of a decimal point. */ #define EXTRACT_DIGIT(s, x, d, dot) \ if (s == dot) *s++ = '.'; *s++ = '0' + (char)(x / d); x %= d +#if defined(__GNUC__) && !defined(__INTEL_COMPILER) && __GNUC__ >= 12 + #pragma GCC diagnostic push + #pragma GCC diagnostic ignored "-Wstringop-overflow" +#endif static inline char * word_to_string(char *s, mpd_uint_t x, int n, char *dot) { @@ -378,6 +382,9 @@ word_to_string(char *s, mpd_uint_t x, int n, char *dot) *s = '\0'; return s; } +#if defined(__GNUC__) && !defined(__INTEL_COMPILER) && __GNUC__ >= 12 + #pragma GCC diagnostic pop +#endif /* Print exponent x to string s. Undefined for MPD_SSIZE_MIN. */ static inline char *

1 0

[3.13] gh-129438: Update ``--enable-experimental-jit`` section with install requirements (GH-129450) (#129472)
by Eclips4 Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/fcb265671c0443133ad9a4f7bd4a1e4fa5… commit: fcb265671c0443133ad9a4f7bd4a1e4fa5810270 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Eclips4 <kirill.bast9(a)mail.ru> date: 2025-01-30T10:22:05Z summary: [3.13] gh-129438: Update ``--enable-experimental-jit`` section with install requirements (GH-129450) (#129472) gh-129438: Update ``--enable-experimental-jit`` section with install requirements (GH-129450) Add a note to the `JIT` docs that building CPython with `JIT` requires Python 3.11 or newer. (cherry picked from commit 652f66ac386dad5992c6f7c494871907272503f8) Co-authored-by: Stan Ulbrych <89152624+StanFromIreland(a)users.noreply.github.com> Co-authored-by: Brandt Bucher <brandtbucher(a)gmail.com> files: M Doc/using/configure.rst M Tools/jit/README.md diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index bf980b514ccd96..1a1f8ae02df0c0 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -314,6 +314,10 @@ General Options By convention, ``--enable-experimental-jit`` is a shorthand for ``--enable-experimental-jit=yes``. + .. note:: + + When building CPython with JIT enabled, ensure that your system has Python 3.11 or later installed. + .. versionadded:: 3.13 .. option:: PKG_CONFIG diff --git a/Tools/jit/README.md b/Tools/jit/README.md index bc6f793b296f12..42d4fcd48e6b41 100644 --- a/Tools/jit/README.md +++ b/Tools/jit/README.md @@ -3,6 +3,8 @@ The JIT Compiler This version of CPython can be built with an experimental just-in-time compiler[^pep-744]. While most everything you already know about building and using CPython is unchanged, you will probably need to install a compatible version of LLVM first. +Python 3.11 or newer is required to build the JIT. + ## Installing LLVM The JIT compiler does not require end users to install any third-party dependencies, but part of it must be *built* using LLVM[^why-llvm]. You are *not* required to build the rest of CPython using LLVM, or even the same version of LLVM (in fact, this is uncommon). @@ -57,7 +59,7 @@ For `PCbuild`-based builds, pass the new `--experimental-jit` option to `build.b For all other builds, pass the new `--enable-experimental-jit` option to `configure`. -Otherwise, just configure and build as you normally would. Cross-compiling "just works", since the JIT is built for the host platform. +Otherwise, just configure and build as you normally would. Cross-compiling "just works", since the JIT is built for the host platform. The JIT can also be enabled or disabled using the `PYTHON_JIT` environment variable, even on builds where it is enabled or disabled by default. More details about configuring CPython with the JIT and optional values for `--enable-experimental-jit` can be found [here](https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compi….

1 0

gh-129438: Update ``--enable-experimental-jit`` section with install requirements (#129450)
by Eclips4 Jan. 30, 2025

Jan. 30, 2025

https://github.com/python/cpython/commit/652f66ac386dad5992c6f7c49487190727… commit: 652f66ac386dad5992c6f7c494871907272503f8 branch: main author: Stan Ulbrych <89152624+StanFromIreland(a)users.noreply.github.com> committer: Eclips4 <kirill.bast9(a)mail.ru> date: 2025-01-30T12:16:05+02:00 summary: gh-129438: Update ``--enable-experimental-jit`` section with install requirements (#129450) Add a note to the `JIT` docs that building CPython with `JIT` requires Python 3.11 or newer. Co-authored-by: Brandt Bucher <brandtbucher(a)gmail.com> files: M Doc/using/configure.rst M Tools/jit/README.md diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index b58532acc4b322..629859e36cb654 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -311,6 +311,10 @@ General Options By convention, ``--enable-experimental-jit`` is a shorthand for ``--enable-experimental-jit=yes``. + .. note:: + + When building CPython with JIT enabled, ensure that your system has Python 3.11 or later installed. + .. versionadded:: 3.13 .. option:: PKG_CONFIG diff --git a/Tools/jit/README.md b/Tools/jit/README.md index 801c64e4059ccc..4107265754f6ec 100644 --- a/Tools/jit/README.md +++ b/Tools/jit/README.md @@ -3,6 +3,8 @@ The JIT Compiler This version of CPython can be built with an experimental just-in-time compiler[^pep-744]. While most everything you already know about building and using CPython is unchanged, you will probably need to install a compatible version of LLVM first. +Python 3.11 or newer is required to build the JIT. + ## Installing LLVM The JIT compiler does not require end users to install any third-party dependencies, but part of it must be *built* using LLVM[^why-llvm]. You are *not* required to build the rest of CPython using LLVM, or even the same version of LLVM (in fact, this is uncommon). @@ -54,7 +56,7 @@ For `PCbuild`-based builds, pass the new `--experimental-jit` option to `build.b For all other builds, pass the new `--enable-experimental-jit` option to `configure`. -Otherwise, just configure and build as you normally would. Cross-compiling "just works", since the JIT is built for the host platform. +Otherwise, just configure and build as you normally would. Cross-compiling "just works", since the JIT is built for the host platform. The JIT can also be enabled or disabled using the `PYTHON_JIT` environment variable, even on builds where it is enabled or disabled by default. More details about configuring CPython with the JIT and optional values for `--enable-experimental-jit` can be found [here](https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compi….

1 0