Mailman 3 June 2024 - Python-checkins

[3.13] gh-120373: Mark test_audit.test_http as requiring the network resource (GH-120374) (#120948)
by carljm 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/502a99a7e66adc095842addfb366bba695… commit: 502a99a7e66adc095842addfb366bba6953f34a1 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: carljm <carl(a)oddbird.net> date: 2024-06-24T13:44:06Z summary: [3.13] gh-120373: Mark test_audit.test_http as requiring the network resource (GH-120374) (#120948) gh-120373: Mark test_audit.test_http as requiring the network resource (GH-120374) (cherry picked from commit b0e1c51882e3a129d1e4db8291f7a0d869d6f1d6) Co-authored-by: Itamar Oren <itamarost(a)gmail.com> files: M Lib/test/test_audit.py diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index 321d4f9abce8c7..7206307d8b0664 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -140,6 +140,7 @@ def test_gc(self): ) + @support.requires_resource('network') def test_http(self): import_helper.import_module("http.client") returncode, events, stderr = self.run_python("test_http_client")

1 0

[3.12] gh-120373: Mark test_audit.test_http as requiring the network resource (GH-120374) (#120949)
by carljm 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/7a874b4ca6150681994a2a38b988711575… commit: 7a874b4ca6150681994a2a38b988711575f53d36 branch: 3.12 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: carljm <carl(a)oddbird.net> date: 2024-06-24T13:37:51Z summary: [3.12] gh-120373: Mark test_audit.test_http as requiring the network resource (GH-120374) (#120949) gh-120373: Mark test_audit.test_http as requiring the network resource (GH-120374) (cherry picked from commit b0e1c51882e3a129d1e4db8291f7a0d869d6f1d6) Co-authored-by: Itamar Oren <itamarost(a)gmail.com> files: M Lib/test/test_audit.py diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index 9e3e03748da14d..9076448ccfd45f 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -140,6 +140,7 @@ def test_gc(self): ) + @support.requires_resource('network') def test_http(self): import_helper.import_module("http.client") returncode, events, stderr = self.run_python("test_http_client")

1 0

docs: puremagic.what() as replacement for imghdr.what() (#120871)
by vstinner 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/22b8a35d6e6660cf7457ed6636cb8c12ff… commit: 22b8a35d6e6660cf7457ed6636cb8c12fff7e8e7 branch: main author: Christian Clauss <cclauss(a)me.com> committer: vstinner <vstinner(a)python.org> date: 2024-06-24T15:32:13+02:00 summary: docs: puremagic.what() as replacement for imghdr.what() (#120871) files: M Doc/whatsnew/3.13.rst diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst index 02c2a5ce02b730..5c195049980a4b 100644 --- a/Doc/whatsnew/3.13.rst +++ b/Doc/whatsnew/3.13.rst @@ -1307,6 +1307,9 @@ PEP 594: dead batteries (and other module removals) * :mod:`!imghdr`: use the projects :pypi:`filetype`, :pypi:`puremagic`, or :pypi:`python-magic` instead. + The ``puremagic.what()`` function can be used to replace + the ``imghdr.what()`` function for all file formats that + were supported by ``imghdr``. (Contributed by Victor Stinner in :gh:`104773`.) * :mod:`!mailcap`.

1 0

gh-120373: Mark test_audit.test_http as requiring the network resource (#120374)
by carljm 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/b0e1c51882e3a129d1e4db8291f7a0d869… commit: b0e1c51882e3a129d1e4db8291f7a0d869d6f1d6 branch: main author: Itamar Oren <itamarost(a)gmail.com> committer: carljm <carl(a)oddbird.net> date: 2024-06-24T07:18:46-06:00 summary: gh-120373: Mark test_audit.test_http as requiring the network resource (#120374) files: M Lib/test/test_audit.py diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index 321d4f9abce8c7..7206307d8b0664 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -140,6 +140,7 @@ def test_gc(self): ) + @support.requires_resource('network') def test_http(self): import_helper.import_module("http.client") returncode, events, stderr = self.run_python("test_http_client")

1 0

gh-119521: Rename IncompleteInputError to _IncompleteInputError and remove from public API/ABI (GH-119680)
by encukou 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/ac61d58db0753a3b37de21dbc6e86b38f2… commit: ac61d58db0753a3b37de21dbc6e86b38f2a93f1b branch: main author: Pablo Galindo Salgado <Pablogsal(a)gmail.com> committer: encukou <encukou(a)gmail.com> date: 2024-06-24T14:08:12+02:00 summary: gh-119521: Rename IncompleteInputError to _IncompleteInputError and remove from public API/ABI (GH-119680) Signed-off-by: Pablo Galindo <pablogsal(a)gmail.com> Co-authored-by: Petr Viktorin <encukou(a)gmail.com> files: M Doc/data/stable_abi.dat M Include/internal/pycore_pyerrors.h M Include/pyerrors.h M Lib/codeop.py M Lib/test/exception_hierarchy.txt M Lib/test/test_pickle.py M Lib/test/test_stable_abi_ctypes.py M Misc/stable_abi.toml M Objects/exceptions.c M PC/python3dll.c M Parser/pegen.c diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index c18c813104cf65..1f7af436a4150b 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -226,7 +226,6 @@ var,PyExc_GeneratorExit,3.2,, var,PyExc_IOError,3.2,, var,PyExc_ImportError,3.2,, var,PyExc_ImportWarning,3.2,, -var,PyExc_IncompleteInputError,3.13,, var,PyExc_IndentationError,3.2,, var,PyExc_IndexError,3.2,, var,PyExc_InterruptedError,3.7,, diff --git a/Include/internal/pycore_pyerrors.h b/Include/internal/pycore_pyerrors.h index 683d87a0d0b129..1187cbf7e90361 100644 --- a/Include/internal/pycore_pyerrors.h +++ b/Include/internal/pycore_pyerrors.h @@ -167,6 +167,10 @@ void _PyErr_FormatNote(const char *format, ...); Py_DEPRECATED(3.12) extern void _PyErr_ChainExceptions(PyObject *, PyObject *, PyObject *); +// implementation detail for the codeop module. +extern PyTypeObject _PyExc_IncompleteInputError; +#define PyExc_IncompleteInputError ((PyObject *)(&_PyExc_IncompleteInputError)) + #ifdef __cplusplus } #endif diff --git a/Include/pyerrors.h b/Include/pyerrors.h index 68d7985dac8876..5d0028c116e2d8 100644 --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -108,7 +108,6 @@ PyAPI_DATA(PyObject *) PyExc_NotImplementedError; PyAPI_DATA(PyObject *) PyExc_SyntaxError; PyAPI_DATA(PyObject *) PyExc_IndentationError; PyAPI_DATA(PyObject *) PyExc_TabError; -PyAPI_DATA(PyObject *) PyExc_IncompleteInputError; PyAPI_DATA(PyObject *) PyExc_ReferenceError; PyAPI_DATA(PyObject *) PyExc_SystemError; PyAPI_DATA(PyObject *) PyExc_SystemExit; diff --git a/Lib/codeop.py b/Lib/codeop.py index 6ad60e7f85098d..a0276b52d484e3 100644 --- a/Lib/codeop.py +++ b/Lib/codeop.py @@ -65,7 +65,7 @@ def _maybe_compile(compiler, source, filename, symbol): try: compiler(source + "\n", filename, symbol) return None - except IncompleteInputError as e: + except _IncompleteInputError as e: return None except SyntaxError as e: pass diff --git a/Lib/test/exception_hierarchy.txt b/Lib/test/exception_hierarchy.txt index 65f54859e2a21d..5e83faab9a6158 100644 --- a/Lib/test/exception_hierarchy.txt +++ b/Lib/test/exception_hierarchy.txt @@ -45,7 +45,7 @@ BaseException ├── StopAsyncIteration ├── StopIteration ├── SyntaxError - │ └── IncompleteInputError + │ └── _IncompleteInputError │ └── IndentationError │ └── TabError ├── SystemError diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py index 19f977971570b7..49aa4b386039ec 100644 --- a/Lib/test/test_pickle.py +++ b/Lib/test/test_pickle.py @@ -569,7 +569,7 @@ def test_exceptions(self): EncodingWarning, BaseExceptionGroup, ExceptionGroup, - IncompleteInputError): + _IncompleteInputError): continue if exc is not OSError and issubclass(exc, OSError): self.assertEqual(reverse_mapping('builtins', name), diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index 47dff5c28f6ff8..d1d8a967dbe62f 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -267,7 +267,6 @@ def test_windows_feature_macros(self): "PyExc_IOError", "PyExc_ImportError", "PyExc_ImportWarning", - "PyExc_IncompleteInputError", "PyExc_IndentationError", "PyExc_IndexError", "PyExc_InterruptedError", diff --git a/Misc/stable_abi.toml b/Misc/stable_abi.toml index 305978f9f0c5c4..73012193d61485 100644 --- a/Misc/stable_abi.toml +++ b/Misc/stable_abi.toml @@ -2480,8 +2480,6 @@ [function._Py_SetRefcnt] added = '3.13' abi_only = true -[data.PyExc_IncompleteInputError] - added = '3.13' [function.PyList_GetItemRef] added = '3.13' [typedef.PyCFunctionFast] diff --git a/Objects/exceptions.c b/Objects/exceptions.c index 3a72cce1dff0c7..6376f2f012a7d6 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -545,10 +545,10 @@ static PyTypeObject _PyExc_ ## EXCNAME = { \ }; \ PyObject *PyExc_ ## EXCNAME = (PyObject *)&_PyExc_ ## EXCNAME -#define MiddlingExtendsException(EXCBASE, EXCNAME, EXCSTORE, EXCDOC) \ -static PyTypeObject _PyExc_ ## EXCNAME = { \ +#define MiddlingExtendsExceptionEx(EXCBASE, EXCNAME, PYEXCNAME, EXCSTORE, EXCDOC) \ +PyTypeObject _PyExc_ ## EXCNAME = { \ PyVarObject_HEAD_INIT(NULL, 0) \ - # EXCNAME, \ + # PYEXCNAME, \ sizeof(Py ## EXCSTORE ## Object), \ 0, (destructor)EXCSTORE ## _dealloc, 0, 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, \ @@ -557,8 +557,12 @@ static PyTypeObject _PyExc_ ## EXCNAME = { \ (inquiry)EXCSTORE ## _clear, 0, 0, 0, 0, 0, 0, 0, &_ ## EXCBASE, \ 0, 0, 0, offsetof(Py ## EXCSTORE ## Object, dict), \ (initproc)EXCSTORE ## _init, 0, 0, \ -}; \ -PyObject *PyExc_ ## EXCNAME = (PyObject *)&_PyExc_ ## EXCNAME +}; + +#define MiddlingExtendsException(EXCBASE, EXCNAME, EXCSTORE, EXCDOC) \ + static MiddlingExtendsExceptionEx( \ + EXCBASE, EXCNAME, EXCNAME, EXCSTORE, EXCDOC); \ + PyObject *PyExc_ ## EXCNAME = (PyObject *)&_PyExc_ ## EXCNAME #define ComplexExtendsException(EXCBASE, EXCNAME, EXCSTORE, EXCNEW, \ EXCMETHODS, EXCMEMBERS, EXCGETSET, \ @@ -2608,8 +2612,8 @@ MiddlingExtendsException(PyExc_IndentationError, TabError, SyntaxError, /* * IncompleteInputError extends SyntaxError */ -MiddlingExtendsException(PyExc_SyntaxError, IncompleteInputError, SyntaxError, - "incomplete input."); +MiddlingExtendsExceptionEx(PyExc_SyntaxError, IncompleteInputError, _IncompleteInputError, + SyntaxError, "incomplete input."); /* * LookupError extends Exception @@ -3675,7 +3679,7 @@ static struct static_exception static_exceptions[] = { // Level 4: Other subclasses ITEM(IndentationError), // base: SyntaxError(Exception) - ITEM(IncompleteInputError), // base: SyntaxError(Exception) + {&_PyExc_IncompleteInputError, "_IncompleteInputError"}, // base: SyntaxError(Exception) ITEM(IndexError), // base: LookupError(Exception) ITEM(KeyError), // base: LookupError(Exception) ITEM(ModuleNotFoundError), // base: ImportError(Exception) diff --git a/PC/python3dll.c b/PC/python3dll.c index 0bcf1cc507e1e8..aa3c3965908ff4 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -840,7 +840,6 @@ EXPORT_DATA(PyExc_FutureWarning) EXPORT_DATA(PyExc_GeneratorExit) EXPORT_DATA(PyExc_ImportError) EXPORT_DATA(PyExc_ImportWarning) -EXPORT_DATA(PyExc_IncompleteInputError) EXPORT_DATA(PyExc_IndentationError) EXPORT_DATA(PyExc_IndexError) EXPORT_DATA(PyExc_InterruptedError) diff --git a/Parser/pegen.c b/Parser/pegen.c index 57a7acea1047b2..6efb5477c7b80f 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -1,6 +1,7 @@ #include <Python.h> #include "pycore_ast.h" // _PyAST_Validate(), #include "pycore_pystate.h" // _PyThreadState_GET() +#include "pycore_pyerrors.h" // PyExc_IncompleteInputError #include <errcode.h> #include "lexer/lexer.h"

1 0

[3.13] gh-119614: Fix truncation of strings with embedded null characters in Tkinter (GH-120909) (GH-120938)
by serhiy-storchaka 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/732c00550f20e73edeac03d9e222c4a719… commit: 732c00550f20e73edeac03d9e222c4a719362649 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2024-06-24T09:45:45Z summary: [3.13] gh-119614: Fix truncation of strings with embedded null characters in Tkinter (GH-120909) (GH-120938) Now the null character is always represented as \xc0\x80 for Tcl_NewStringObj(). (cherry picked from commit c38e2f64d012929168dfef7363c9e48bd1a6c731) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: A Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst M Lib/test/test_tcl.py M Lib/test/test_tkinter/test_misc.py M Modules/_tkinter.c diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index 443787d721d5fb..d479f7d7515d9b 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -73,6 +73,18 @@ def testCall(self): tcl.call('set','a','1') self.assertEqual(tcl.call('set','a'),'1') + def test_call_passing_null(self): + tcl = self.interp + tcl.call('set', 'a', 'a\0b') # ASCII-only + self.assertEqual(tcl.getvar('a'), 'a\x00b') + self.assertEqual(tcl.call('set', 'a'), 'a\x00b') + self.assertEqual(tcl.eval('set a'), 'a\x00b') + + tcl.call('set', 'a', '\u20ac\0') # non-ASCII + self.assertEqual(tcl.getvar('a'), '\u20ac\x00') + self.assertEqual(tcl.call('set', 'a'), '\u20ac\x00') + self.assertEqual(tcl.eval('set a'), '\u20ac\x00') + def testCallException(self): tcl = self.interp self.assertRaises(TclError,tcl.call,'set','a') @@ -98,6 +110,18 @@ def testSetVar(self): tcl.setvar('a','1') self.assertEqual(tcl.eval('set a'),'1') + def test_setvar_passing_null(self): + tcl = self.interp + tcl.setvar('a', 'a\0b') # ASCII-only + self.assertEqual(tcl.getvar('a'), 'a\x00b') + self.assertEqual(tcl.call('set', 'a'), 'a\x00b') + self.assertEqual(tcl.eval('set a'), 'a\x00b') + + tcl.setvar('a', '\u20ac\0') # non-ASCII + self.assertEqual(tcl.getvar('a'), '\u20ac\x00') + self.assertEqual(tcl.call('set', 'a'), '\u20ac\x00') + self.assertEqual(tcl.eval('set a'), '\u20ac\x00') + def testSetVarArray(self): tcl = self.interp tcl.setvar('a(1)','1') diff --git a/Lib/test/test_tkinter/test_misc.py b/Lib/test/test_tkinter/test_misc.py index d9ea642881a179..b0b9ed60040443 100644 --- a/Lib/test/test_tkinter/test_misc.py +++ b/Lib/test/test_tkinter/test_misc.py @@ -476,6 +476,15 @@ def test_info_patchlevel(self): self.assertEqual(vi.micro, 0) self.assertTrue(str(vi).startswith(f'{vi.major}.{vi.minor}')) + def test_embedded_null(self): + widget = tkinter.Entry(self.root) + widget.insert(0, 'abc\0def') # ASCII-only + widget.selection_range(0, 'end') + self.assertEqual(widget.selection_get(), 'abc\x00def') + widget.insert(0, '\u20ac\0') # non-ASCII + widget.selection_range(0, 'end') + self.assertEqual(widget.selection_get(), '\u20ac\0abc\x00def') + class WmTest(AbstractTkTest, unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst b/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst new file mode 100644 index 00000000000000..d518265a7fe55a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst @@ -0,0 +1,2 @@ +Fix truncation of strings with embedded null characters in some internal +operations in :mod:`tkinter`. diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index 8fe2c5ba295148..cd3722f54c24ce 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -512,7 +512,7 @@ unicodeFromTclObj(TkappObject *tkapp, Tcl_Obj *value) else Py_UNREACHABLE(); } -#endif +#endif /* USE_TCL_UNICODE */ const char *s = Tcl_GetStringFromObj(value, &len); return unicodeFromTclStringAndSize(s, len); } @@ -1018,7 +1018,9 @@ AsObj(PyObject *value) PyErr_SetString(PyExc_OverflowError, "string is too long"); return NULL; } - if (PyUnicode_IS_ASCII(value)) { + if (PyUnicode_IS_ASCII(value) && + strlen(PyUnicode_DATA(value)) == (size_t)PyUnicode_GET_LENGTH(value)) + { return Tcl_NewStringObj((const char *)PyUnicode_DATA(value), (int)size); } @@ -1033,9 +1035,6 @@ AsObj(PyObject *value) "surrogatepass", NATIVE_BYTEORDER); else Py_UNREACHABLE(); -#else - encoded = _PyUnicode_AsUTF8String(value, "surrogateescape"); -#endif if (!encoded) { return NULL; } @@ -1045,12 +1044,39 @@ AsObj(PyObject *value) PyErr_SetString(PyExc_OverflowError, "string is too long"); return NULL; } -#if USE_TCL_UNICODE result = Tcl_NewUnicodeObj((const Tcl_UniChar *)PyBytes_AS_STRING(encoded), (int)(size / sizeof(Tcl_UniChar))); #else + encoded = _PyUnicode_AsUTF8String(value, "surrogateescape"); + if (!encoded) { + return NULL; + } + size = PyBytes_GET_SIZE(encoded); + if (strlen(PyBytes_AS_STRING(encoded)) != (size_t)size) { + /* The string contains embedded null characters. + * Tcl needs a null character to be represented as \xc0\x80 in + * the Modified UTF-8 encoding. Otherwise the string can be + * truncated in some internal operations. + * + * NOTE: stringlib_replace() could be used here, but optimizing + * this obscure case isn't worth it unless stringlib_replace() + * was already exposed in the C API for other reasons. */ + Py_SETREF(encoded, + PyObject_CallMethod(encoded, "replace", "y#y#", + "\0", (Py_ssize_t)1, + "\xc0\x80", (Py_ssize_t)2)); + if (!encoded) { + return NULL; + } + size = PyBytes_GET_SIZE(encoded); + } + if (size > INT_MAX) { + Py_DECREF(encoded); + PyErr_SetString(PyExc_OverflowError, "string is too long"); + return NULL; + } result = Tcl_NewStringObj(PyBytes_AS_STRING(encoded), (int)size); -#endif +#endif /* USE_TCL_UNICODE */ Py_DECREF(encoded); return result; }

1 0

[3.12] gh-119614: Fix truncation of strings with embedded null characters in Tkinter (GH-120909) (GH-120939)
by serhiy-storchaka 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/d5a8d4b19670b930cd6cb5e18e267877eb… commit: d5a8d4b19670b930cd6cb5e18e267877ebe49233 branch: 3.12 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2024-06-24T09:39:56Z summary: [3.12] gh-119614: Fix truncation of strings with embedded null characters in Tkinter (GH-120909) (GH-120939) Now the null character is always represented as \xc0\x80 for Tcl_NewStringObj(). (cherry picked from commit c38e2f64d012929168dfef7363c9e48bd1a6c731) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: A Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst M Lib/test/test_tcl.py M Lib/test/test_tkinter/test_misc.py M Modules/_tkinter.c diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index 55a1fa66914c36..743ff85dd19075 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -73,6 +73,18 @@ def testCall(self): tcl.call('set','a','1') self.assertEqual(tcl.call('set','a'),'1') + def test_call_passing_null(self): + tcl = self.interp + tcl.call('set', 'a', 'a\0b') # ASCII-only + self.assertEqual(tcl.getvar('a'), 'a\x00b') + self.assertEqual(tcl.call('set', 'a'), 'a\x00b') + self.assertEqual(tcl.eval('set a'), 'a\x00b') + + tcl.call('set', 'a', '\u20ac\0') # non-ASCII + self.assertEqual(tcl.getvar('a'), '\u20ac\x00') + self.assertEqual(tcl.call('set', 'a'), '\u20ac\x00') + self.assertEqual(tcl.eval('set a'), '\u20ac\x00') + def testCallException(self): tcl = self.interp self.assertRaises(TclError,tcl.call,'set','a') @@ -98,6 +110,18 @@ def testSetVar(self): tcl.setvar('a','1') self.assertEqual(tcl.eval('set a'),'1') + def test_setvar_passing_null(self): + tcl = self.interp + tcl.setvar('a', 'a\0b') # ASCII-only + self.assertEqual(tcl.getvar('a'), 'a\x00b') + self.assertEqual(tcl.call('set', 'a'), 'a\x00b') + self.assertEqual(tcl.eval('set a'), 'a\x00b') + + tcl.setvar('a', '\u20ac\0') # non-ASCII + self.assertEqual(tcl.getvar('a'), '\u20ac\x00') + self.assertEqual(tcl.call('set', 'a'), '\u20ac\x00') + self.assertEqual(tcl.eval('set a'), '\u20ac\x00') + def testSetVarArray(self): tcl = self.interp tcl.setvar('a(1)','1') diff --git a/Lib/test/test_tkinter/test_misc.py b/Lib/test/test_tkinter/test_misc.py index fc12860010efba..81866993435a0c 100644 --- a/Lib/test/test_tkinter/test_misc.py +++ b/Lib/test/test_tkinter/test_misc.py @@ -382,6 +382,15 @@ def test_info_patchlevel(self): self.assertEqual(vi.micro, 0) self.assertTrue(str(vi).startswith(f'{vi.major}.{vi.minor}')) + def test_embedded_null(self): + widget = tkinter.Entry(self.root) + widget.insert(0, 'abc\0def') # ASCII-only + widget.selection_range(0, 'end') + self.assertEqual(widget.selection_get(), 'abc\x00def') + widget.insert(0, '\u20ac\0') # non-ASCII + widget.selection_range(0, 'end') + self.assertEqual(widget.selection_get(), '\u20ac\0abc\x00def') + class EventTest(AbstractTkTest, unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst b/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst new file mode 100644 index 00000000000000..d518265a7fe55a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst @@ -0,0 +1,2 @@ +Fix truncation of strings with embedded null characters in some internal +operations in :mod:`tkinter`. diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index 29a3d28c090c43..6b5fcb8a365b61 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -516,7 +516,7 @@ unicodeFromTclObj(TkappObject *tkapp, Tcl_Obj *value) else Py_UNREACHABLE(); } -#endif +#endif /* USE_TCL_UNICODE */ const char *s = Tcl_GetStringFromObj(value, &len); return unicodeFromTclStringAndSize(s, len); } @@ -1024,7 +1024,9 @@ AsObj(PyObject *value) PyErr_SetString(PyExc_OverflowError, "string is too long"); return NULL; } - if (PyUnicode_IS_ASCII(value)) { + if (PyUnicode_IS_ASCII(value) && + strlen(PyUnicode_DATA(value)) == (size_t)PyUnicode_GET_LENGTH(value)) + { return Tcl_NewStringObj((const char *)PyUnicode_DATA(value), (int)size); } @@ -1039,9 +1041,6 @@ AsObj(PyObject *value) "surrogatepass", NATIVE_BYTEORDER); else Py_UNREACHABLE(); -#else - encoded = _PyUnicode_AsUTF8String(value, "surrogateescape"); -#endif if (!encoded) { return NULL; } @@ -1051,12 +1050,39 @@ AsObj(PyObject *value) PyErr_SetString(PyExc_OverflowError, "string is too long"); return NULL; } -#if USE_TCL_UNICODE result = Tcl_NewUnicodeObj((const Tcl_UniChar *)PyBytes_AS_STRING(encoded), (int)(size / sizeof(Tcl_UniChar))); #else + encoded = _PyUnicode_AsUTF8String(value, "surrogateescape"); + if (!encoded) { + return NULL; + } + size = PyBytes_GET_SIZE(encoded); + if (strlen(PyBytes_AS_STRING(encoded)) != (size_t)size) { + /* The string contains embedded null characters. + * Tcl needs a null character to be represented as \xc0\x80 in + * the Modified UTF-8 encoding. Otherwise the string can be + * truncated in some internal operations. + * + * NOTE: stringlib_replace() could be used here, but optimizing + * this obscure case isn't worth it unless stringlib_replace() + * was already exposed in the C API for other reasons. */ + Py_SETREF(encoded, + PyObject_CallMethod(encoded, "replace", "y#y#", + "\0", (Py_ssize_t)1, + "\xc0\x80", (Py_ssize_t)2)); + if (!encoded) { + return NULL; + } + size = PyBytes_GET_SIZE(encoded); + } + if (size > INT_MAX) { + Py_DECREF(encoded); + PyErr_SetString(PyExc_OverflowError, "string is too long"); + return NULL; + } result = Tcl_NewStringObj(PyBytes_AS_STRING(encoded), (int)size); -#endif +#endif /* USE_TCL_UNICODE */ Py_DECREF(encoded); return result; }

1 0

gh-120834: fix type of *_iframe field in _PyGenObject_HEAD declaration (#120835)
by iritkatriel 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/65a12c559cbc13c2c5a4aa65c76310bd8d… commit: 65a12c559cbc13c2c5a4aa65c76310bd8d2051a7 branch: main author: Irit Katriel <1055913+iritkatriel(a)users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel(a)users.noreply.github.com> date: 2024-06-24T10:23:38+01:00 summary: gh-120834: fix type of *_iframe field in _PyGenObject_HEAD declaration (#120835) files: M Include/cpython/genobject.h M Include/internal/pycore_frame.h M Include/internal/pycore_genobject.h M Include/internal/pycore_interp.h M Objects/frameobject.c M Objects/genobject.c M Python/bytecodes.c M Python/ceval.c M Python/executor_cases.c.h M Python/frame.c M Python/generated_cases.c.h M Python/pylifecycle.c diff --git a/Include/cpython/genobject.h b/Include/cpython/genobject.h index 49e46c277d75ae..f75884e597e2c2 100644 --- a/Include/cpython/genobject.h +++ b/Include/cpython/genobject.h @@ -9,29 +9,7 @@ extern "C" { /* --- Generators --------------------------------------------------------- */ -/* _PyGenObject_HEAD defines the initial segment of generator - and coroutine objects. */ -#define _PyGenObject_HEAD(prefix) \ - PyObject_HEAD \ - /* List of weak reference. */ \ - PyObject *prefix##_weakreflist; \ - /* Name of the generator. */ \ - PyObject *prefix##_name; \ - /* Qualified name of the generator. */ \ - PyObject *prefix##_qualname; \ - _PyErr_StackItem prefix##_exc_state; \ - PyObject *prefix##_origin_or_finalizer; \ - char prefix##_hooks_inited; \ - char prefix##_closed; \ - char prefix##_running_async; \ - /* The frame */ \ - int8_t prefix##_frame_state; \ - PyObject *prefix##_iframe[1]; \ - -typedef struct { - /* The gi_ prefix is intended to remind of generator-iterator. */ - _PyGenObject_HEAD(gi) -} PyGenObject; +typedef struct _PyGenObject PyGenObject; PyAPI_DATA(PyTypeObject) PyGen_Type; @@ -46,9 +24,7 @@ PyAPI_FUNC(PyCodeObject *) PyGen_GetCode(PyGenObject *gen); /* --- PyCoroObject ------------------------------------------------------- */ -typedef struct { - _PyGenObject_HEAD(cr) -} PyCoroObject; +typedef struct _PyCoroObject PyCoroObject; PyAPI_DATA(PyTypeObject) PyCoro_Type; @@ -59,9 +35,7 @@ PyAPI_FUNC(PyObject *) PyCoro_New(PyFrameObject *, /* --- Asynchronous Generators -------------------------------------------- */ -typedef struct { - _PyGenObject_HEAD(ag) -} PyAsyncGenObject; +typedef struct _PyAsyncGenObject PyAsyncGenObject; PyAPI_DATA(PyTypeObject) PyAsyncGen_Type; PyAPI_DATA(PyTypeObject) _PyAsyncGenASend_Type; @@ -73,7 +47,6 @@ PyAPI_FUNC(PyObject *) PyAsyncGen_New(PyFrameObject *, #define PyAsyncGenASend_CheckExact(op) Py_IS_TYPE((op), &_PyAsyncGenASend_Type) - #undef _PyGenObject_HEAD #ifdef __cplusplus diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index e4eb893263c42c..bab92c771a76b1 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -307,14 +307,6 @@ _PyFrame_PushTrampolineUnchecked(PyThreadState *tstate, PyCodeObject *code, int return frame; } -static inline -PyGenObject *_PyFrame_GetGenerator(_PyInterpreterFrame *frame) -{ - assert(frame->owner == FRAME_OWNED_BY_GENERATOR); - size_t offset_in_gen = offsetof(PyGenObject, gi_iframe); - return (PyGenObject *)(((char *)frame) - offset_in_gen); -} - PyAPI_FUNC(_PyInterpreterFrame *) _PyEvalFramePushAndInit(PyThreadState *tstate, PyFunctionObject *func, PyObject *locals, PyObject* const* args, diff --git a/Include/internal/pycore_genobject.h b/Include/internal/pycore_genobject.h index 9463c822ad8669..f6d7e6d367177b 100644 --- a/Include/internal/pycore_genobject.h +++ b/Include/internal/pycore_genobject.h @@ -8,7 +8,49 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif -#include "pycore_freelist.h" +#include "pycore_frame.h" + +/* _PyGenObject_HEAD defines the initial segment of generator + and coroutine objects. */ +#define _PyGenObject_HEAD(prefix) \ + PyObject_HEAD \ + /* List of weak reference. */ \ + PyObject *prefix##_weakreflist; \ + /* Name of the generator. */ \ + PyObject *prefix##_name; \ + /* Qualified name of the generator. */ \ + PyObject *prefix##_qualname; \ + _PyErr_StackItem prefix##_exc_state; \ + PyObject *prefix##_origin_or_finalizer; \ + char prefix##_hooks_inited; \ + char prefix##_closed; \ + char prefix##_running_async; \ + /* The frame */ \ + int8_t prefix##_frame_state; \ + struct _PyInterpreterFrame prefix##_iframe; \ + +struct _PyGenObject { + /* The gi_ prefix is intended to remind of generator-iterator. */ + _PyGenObject_HEAD(gi) +}; + +struct _PyCoroObject { + _PyGenObject_HEAD(cr) +}; + +struct _PyAsyncGenObject { + _PyGenObject_HEAD(ag) +}; + +#undef _PyGenObject_HEAD + +static inline +PyGenObject *_PyGen_GetGeneratorFromFrame(_PyInterpreterFrame *frame) +{ + assert(frame->owner == FRAME_OWNED_BY_GENERATOR); + size_t offset_in_gen = offsetof(PyGenObject, gi_iframe); + return (PyGenObject *)(((char *)frame) - offset_in_gen); +} PyAPI_FUNC(PyObject *)_PyGen_yf(PyGenObject *); extern void _PyGen_Finalize(PyObject *self); diff --git a/Include/internal/pycore_interp.h b/Include/internal/pycore_interp.h index 6b5f50b88f7b85..b40b63a2a25014 100644 --- a/Include/internal/pycore_interp.h +++ b/Include/internal/pycore_interp.h @@ -23,7 +23,7 @@ extern "C" { #include "pycore_floatobject.h" // struct _Py_float_state #include "pycore_function.h" // FUNC_MAX_WATCHERS #include "pycore_gc.h" // struct _gc_runtime_state -#include "pycore_genobject.h" // struct _Py_async_gen_state +#include "pycore_genobject.h" // _PyGen_FetchStopIterationValue #include "pycore_global_objects.h"// struct _Py_interp_cached_objects #include "pycore_import.h" // struct _import_state #include "pycore_instruments.h" // _PY_MONITORING_EVENTS diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 860669cfb7d674..2cb113b3d01be1 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -1338,7 +1338,7 @@ static bool frame_is_suspended(PyFrameObject *frame) { assert(!_PyFrame_IsIncomplete(frame->f_frame)); if (frame->f_frame->owner == FRAME_OWNED_BY_GENERATOR) { - PyGenObject *gen = _PyFrame_GetGenerator(frame->f_frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame->f_frame); return FRAME_STATE_SUSPENDED(gen->gi_frame_state); } return false; @@ -1665,7 +1665,7 @@ static PyObject * frame_clear(PyFrameObject *f, PyObject *Py_UNUSED(ignored)) { if (f->f_frame->owner == FRAME_OWNED_BY_GENERATOR) { - PyGenObject *gen = _PyFrame_GetGenerator(f->f_frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(f->f_frame); if (gen->gi_frame_state == FRAME_EXECUTING) { goto running; } @@ -2097,7 +2097,7 @@ PyFrame_GetGenerator(PyFrameObject *frame) if (frame->f_frame->owner != FRAME_OWNED_BY_GENERATOR) { return NULL; } - PyGenObject *gen = _PyFrame_GetGenerator(frame->f_frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame->f_frame); return Py_NewRef(gen); } diff --git a/Objects/genobject.c b/Objects/genobject.c index 662e644f8aeddd..445622e878bc93 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -6,8 +6,8 @@ #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _PyEval_EvalFrame() #include "pycore_frame.h" // _PyInterpreterFrame +#include "pycore_freelist.h" // struct _Py_async_gen_freelist #include "pycore_gc.h" // _PyGC_CLEAR_FINALIZED() -#include "pycore_genobject.h" // struct _Py_async_gen_freelist #include "pycore_modsupport.h" // _PyArg_CheckPositional() #include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "pycore_opcode_utils.h" // RESUME_AFTER_YIELD_FROM @@ -30,8 +30,7 @@ static const char *ASYNC_GEN_IGNORED_EXIT_MSG = /* Returns a borrowed reference */ static inline PyCodeObject * _PyGen_GetCode(PyGenObject *gen) { - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)(gen->gi_iframe); - return _PyFrame_GetCode(frame); + return _PyFrame_GetCode(&gen->gi_iframe); } PyCodeObject * @@ -48,7 +47,7 @@ gen_traverse(PyGenObject *gen, visitproc visit, void *arg) Py_VISIT(gen->gi_name); Py_VISIT(gen->gi_qualname); if (gen->gi_frame_state != FRAME_CLEARED) { - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)(gen->gi_iframe); + _PyInterpreterFrame *frame = &gen->gi_iframe; assert(frame->frame_obj == NULL || frame->frame_obj->f_frame->owner == FRAME_OWNED_BY_GENERATOR); int err = _PyFrame_Traverse(frame, visit, arg); @@ -141,7 +140,7 @@ gen_dealloc(PyGenObject *gen) Py_CLEAR(((PyAsyncGenObject*)gen)->ag_origin_or_finalizer); } if (gen->gi_frame_state != FRAME_CLEARED) { - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *frame = &gen->gi_iframe; gen->gi_frame_state = FRAME_CLEARED; frame->previous = NULL; _PyFrame_ClearExceptCode(frame); @@ -163,7 +162,7 @@ gen_send_ex2(PyGenObject *gen, PyObject *arg, PyObject **presult, int exc, int closing) { PyThreadState *tstate = _PyThreadState_GET(); - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *frame = &gen->gi_iframe; *presult = NULL; if (gen->gi_frame_state == FRAME_CREATED && arg && arg != Py_None) { @@ -342,7 +341,7 @@ PyObject * _PyGen_yf(PyGenObject *gen) { if (gen->gi_frame_state == FRAME_SUSPENDED_YIELD_FROM) { - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *frame = &gen->gi_iframe; assert(is_resume(frame->instr_ptr)); assert((frame->instr_ptr->op.arg & RESUME_OPARG_LOCATION_MASK) >= RESUME_AFTER_YIELD_FROM); return Py_NewRef(_PyFrame_StackPeek(frame)); @@ -372,7 +371,7 @@ gen_close(PyGenObject *gen, PyObject *args) gen->gi_frame_state = state; Py_DECREF(yf); } - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *frame = &gen->gi_iframe; if (is_resume(frame->instr_ptr)) { /* We can safely ignore the outermost try block * as it is automatically generated to handle @@ -382,7 +381,7 @@ gen_close(PyGenObject *gen, PyObject *args) // RESUME after YIELD_VALUE and exception depth is 1 assert((oparg & RESUME_OPARG_LOCATION_MASK) != RESUME_AT_FUNC_START); gen->gi_frame_state = FRAME_COMPLETED; - _PyFrame_ClearLocals((_PyInterpreterFrame *)gen->gi_iframe); + _PyFrame_ClearLocals(&gen->gi_iframe); Py_RETURN_NONE; } } @@ -431,7 +430,7 @@ _gen_throw(PyGenObject *gen, int close_on_genexit, PyObject *yf = _PyGen_yf(gen); if (yf) { - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *frame = &gen->gi_iframe; PyObject *ret; int err; if (PyErr_GivenExceptionMatches(typ, PyExc_GeneratorExit) && @@ -739,7 +738,7 @@ _gen_getframe(PyGenObject *gen, const char *const name) if (FRAME_STATE_FINISHED(gen->gi_frame_state)) { Py_RETURN_NONE; } - return _Py_XNewRef((PyObject *)_PyFrame_GetFrameObject((_PyInterpreterFrame *)gen->gi_iframe)); + return _Py_XNewRef((PyObject *)_PyFrame_GetFrameObject(&gen->gi_iframe)); } static PyObject * @@ -814,8 +813,7 @@ static PyAsyncMethods gen_as_async = { PyTypeObject PyGen_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "generator", /* tp_name */ - offsetof(PyGenObject, gi_iframe) + - offsetof(_PyInterpreterFrame, localsplus), /* tp_basicsize */ + sizeof(PyGenObject), /* tp_basicsize */ sizeof(PyObject *), /* tp_itemsize */ /* methods */ (destructor)gen_dealloc, /* tp_dealloc */ @@ -949,7 +947,7 @@ gen_new_with_qualname(PyTypeObject *type, PyFrameObject *f, /* Copy the frame */ assert(f->f_frame->frame_obj == NULL); assert(f->f_frame->owner == FRAME_OWNED_BY_FRAME_OBJECT); - _PyInterpreterFrame *frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *frame = &gen->gi_iframe; _PyFrame_Copy((_PyInterpreterFrame *)f->_f_frame_data, frame); gen->gi_frame_state = FRAME_CREATED; assert(frame->frame_obj == f); @@ -1166,8 +1164,7 @@ static PyAsyncMethods coro_as_async = { PyTypeObject PyCoro_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "coroutine", /* tp_name */ - offsetof(PyCoroObject, cr_iframe) + - offsetof(_PyInterpreterFrame, localsplus), /* tp_basicsize */ + sizeof(PyCoroObject), /* tp_basicsize */ sizeof(PyObject *), /* tp_itemsize */ /* methods */ (destructor)gen_dealloc, /* tp_dealloc */ @@ -1582,8 +1579,7 @@ static PyAsyncMethods async_gen_as_async = { PyTypeObject PyAsyncGen_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "async_generator", /* tp_name */ - offsetof(PyAsyncGenObject, ag_iframe) + - offsetof(_PyInterpreterFrame, localsplus), /* tp_basicsize */ + sizeof(PyAsyncGenObject), /* tp_basicsize */ sizeof(PyObject *), /* tp_itemsize */ /* methods */ (destructor)gen_dealloc, /* tp_dealloc */ diff --git a/Python/bytecodes.c b/Python/bytecodes.c index 50444bcc0d200c..a6fb862a6d476e 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -1015,7 +1015,7 @@ dummy_func( ((PyGenObject *)receiver)->gi_frame_state < FRAME_EXECUTING) { PyGenObject *gen = (PyGenObject *)receiver; - _PyInterpreterFrame *gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *gen_frame = &gen->gi_iframe; STACK_SHRINK(1); _PyFrame_StackPush(gen_frame, v); gen->gi_frame_state = FRAME_EXECUTING; @@ -1055,7 +1055,7 @@ dummy_func( DEOPT_IF(Py_TYPE(gen) != &PyGen_Type && Py_TYPE(gen) != &PyCoro_Type); DEOPT_IF(gen->gi_frame_state >= FRAME_EXECUTING); STAT_INC(SEND, hit); - _PyInterpreterFrame *gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *gen_frame = &gen->gi_iframe; STACK_SHRINK(1); _PyFrame_StackPush(gen_frame, v); gen->gi_frame_state = FRAME_EXECUTING; @@ -1069,7 +1069,7 @@ dummy_func( inst(INSTRUMENTED_YIELD_VALUE, (retval -- unused)) { assert(frame != &entry_frame); frame->instr_ptr = next_instr; - PyGenObject *gen = _PyFrame_GetGenerator(frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame); assert(FRAME_SUSPENDED_YIELD_FROM == FRAME_SUSPENDED + 1); assert(oparg == 0 || oparg == 1); gen->gi_frame_state = FRAME_SUSPENDED + oparg; @@ -1099,7 +1099,7 @@ dummy_func( assert(frame != &entry_frame); #endif frame->instr_ptr++; - PyGenObject *gen = _PyFrame_GetGenerator(frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame); assert(FRAME_SUSPENDED_YIELD_FROM == FRAME_SUSPENDED + 1); assert(oparg == 0 || oparg == 1); gen->gi_frame_state = FRAME_SUSPENDED + oparg; @@ -2799,7 +2799,7 @@ dummy_func( DEOPT_IF(Py_TYPE(gen) != &PyGen_Type); DEOPT_IF(gen->gi_frame_state >= FRAME_EXECUTING); STAT_INC(FOR_ITER, hit); - gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + gen_frame = &gen->gi_iframe; _PyFrame_StackPush(gen_frame, Py_None); gen->gi_frame_state = FRAME_EXECUTING; gen->gi_exc_state.previous_item = tstate->exc_info; @@ -3940,7 +3940,7 @@ dummy_func( } assert(EMPTY()); _PyFrame_SetStackPointer(frame, stack_pointer); - _PyInterpreterFrame *gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *gen_frame = &gen->gi_iframe; frame->instr_ptr++; _PyFrame_Copy(frame, gen_frame); assert(frame->frame_obj == NULL); diff --git a/Python/ceval.c b/Python/ceval.c index a8df045799d3fd..d36e15e55f57ab 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1711,7 +1711,7 @@ static void clear_gen_frame(PyThreadState *tstate, _PyInterpreterFrame * frame) { assert(frame->owner == FRAME_OWNED_BY_GENERATOR); - PyGenObject *gen = _PyFrame_GetGenerator(frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame); gen->gi_frame_state = FRAME_CLEARED; assert(tstate->exc_info == &gen->gi_exc_state); tstate->exc_info = gen->gi_exc_state.previous_item; diff --git a/Python/executor_cases.c.h b/Python/executor_cases.c.h index b4e5261f57ab8b..cdfffcdec9726e 100644 --- a/Python/executor_cases.c.h +++ b/Python/executor_cases.c.h @@ -1125,7 +1125,7 @@ assert(frame != &entry_frame); #endif frame->instr_ptr++; - PyGenObject *gen = _PyFrame_GetGenerator(frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame); assert(FRAME_SUSPENDED_YIELD_FROM == FRAME_SUSPENDED + 1); assert(oparg == 0 || oparg == 1); gen->gi_frame_state = FRAME_SUSPENDED + oparg; @@ -2871,7 +2871,7 @@ JUMP_TO_JUMP_TARGET(); } STAT_INC(FOR_ITER, hit); - gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + gen_frame = &gen->gi_iframe; _PyFrame_StackPush(gen_frame, Py_None); gen->gi_frame_state = FRAME_EXECUTING; gen->gi_exc_state.previous_item = tstate->exc_info; @@ -4110,7 +4110,7 @@ } assert(EMPTY()); _PyFrame_SetStackPointer(frame, stack_pointer); - _PyInterpreterFrame *gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *gen_frame = &gen->gi_iframe; frame->instr_ptr++; _PyFrame_Copy(frame, gen_frame); assert(frame->frame_obj == NULL); diff --git a/Python/frame.c b/Python/frame.c index 2bb12823572028..7299a395efad28 100644 --- a/Python/frame.c +++ b/Python/frame.c @@ -112,7 +112,7 @@ _PyFrame_ClearExceptCode(_PyInterpreterFrame *frame) /* It is the responsibility of the owning generator/coroutine * to have cleared the enclosing generator, if any. */ assert(frame->owner != FRAME_OWNED_BY_GENERATOR || - _PyFrame_GetGenerator(frame)->gi_frame_state == FRAME_CLEARED); + _PyGen_GetGeneratorFromFrame(frame)->gi_frame_state == FRAME_CLEARED); // GH-99729: Clearing this frame can expose the stack (via finalizers). It's // crucial that this frame has been unlinked, and is no longer visible: assert(_PyThreadState_GET()->current_frame != frame); diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index 3980f9852e6396..327a704467b0ba 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -2747,7 +2747,7 @@ DEOPT_IF(Py_TYPE(gen) != &PyGen_Type, FOR_ITER); DEOPT_IF(gen->gi_frame_state >= FRAME_EXECUTING, FOR_ITER); STAT_INC(FOR_ITER, hit); - gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + gen_frame = &gen->gi_iframe; _PyFrame_StackPush(gen_frame, Py_None); gen->gi_frame_state = FRAME_EXECUTING; gen->gi_exc_state.previous_item = tstate->exc_info; @@ -3461,7 +3461,7 @@ retval = stack_pointer[-1]; assert(frame != &entry_frame); frame->instr_ptr = next_instr; - PyGenObject *gen = _PyFrame_GetGenerator(frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame); assert(FRAME_SUSPENDED_YIELD_FROM == FRAME_SUSPENDED + 1); assert(oparg == 0 || oparg == 1); gen->gi_frame_state = FRAME_SUSPENDED + oparg; @@ -5180,7 +5180,7 @@ } assert(EMPTY()); _PyFrame_SetStackPointer(frame, stack_pointer); - _PyInterpreterFrame *gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *gen_frame = &gen->gi_iframe; frame->instr_ptr++; _PyFrame_Copy(frame, gen_frame); assert(frame->frame_obj == NULL); @@ -5260,7 +5260,7 @@ ((PyGenObject *)receiver)->gi_frame_state < FRAME_EXECUTING) { PyGenObject *gen = (PyGenObject *)receiver; - _PyInterpreterFrame *gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *gen_frame = &gen->gi_iframe; STACK_SHRINK(1); _PyFrame_StackPush(gen_frame, v); gen->gi_frame_state = FRAME_EXECUTING; @@ -5310,7 +5310,7 @@ DEOPT_IF(Py_TYPE(gen) != &PyGen_Type && Py_TYPE(gen) != &PyCoro_Type, SEND); DEOPT_IF(gen->gi_frame_state >= FRAME_EXECUTING, SEND); STAT_INC(SEND, hit); - _PyInterpreterFrame *gen_frame = (_PyInterpreterFrame *)gen->gi_iframe; + _PyInterpreterFrame *gen_frame = &gen->gi_iframe; STACK_SHRINK(1); _PyFrame_StackPush(gen_frame, v); gen->gi_frame_state = FRAME_EXECUTING; @@ -6198,7 +6198,7 @@ assert(frame != &entry_frame); #endif frame->instr_ptr++; - PyGenObject *gen = _PyFrame_GetGenerator(frame); + PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame); assert(FRAME_SUSPENDED_YIELD_FROM == FRAME_SUSPENDED + 1); assert(oparg == 0 || oparg == 1); gen->gi_frame_state = FRAME_SUSPENDED + oparg; diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 3639cf6712053e..a7b4d7ecc7c3ad 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -10,7 +10,6 @@ #include "pycore_exceptions.h" // _PyExc_InitTypes() #include "pycore_fileutils.h" // _Py_ResetForceASCII() #include "pycore_floatobject.h" // _PyFloat_InitTypes() -#include "pycore_genobject.h" // _PyAsyncGen_Fini() #include "pycore_global_objects_fini_generated.h" // "_PyStaticObjects_CheckRefcnt() #include "pycore_import.h" // _PyImport_BootstrapImp() #include "pycore_initconfig.h" // _PyStatus_OK()

1 0

gh-119614: Fix truncation of strings with embedded null characters in Tkinter (GH-120909)
by serhiy-storchaka 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/c38e2f64d012929168dfef7363c9e48bd1… commit: c38e2f64d012929168dfef7363c9e48bd1a6c731 branch: main author: Serhiy Storchaka <storchaka(a)gmail.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2024-06-24T12:17:25+03:00 summary: gh-119614: Fix truncation of strings with embedded null characters in Tkinter (GH-120909) Now the null character is always represented as \xc0\x80 for Tcl_NewStringObj(). files: A Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst M Lib/test/test_tcl.py M Lib/test/test_tkinter/test_misc.py M Modules/_tkinter.c diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index 443787d721d5fb..d479f7d7515d9b 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -73,6 +73,18 @@ def testCall(self): tcl.call('set','a','1') self.assertEqual(tcl.call('set','a'),'1') + def test_call_passing_null(self): + tcl = self.interp + tcl.call('set', 'a', 'a\0b') # ASCII-only + self.assertEqual(tcl.getvar('a'), 'a\x00b') + self.assertEqual(tcl.call('set', 'a'), 'a\x00b') + self.assertEqual(tcl.eval('set a'), 'a\x00b') + + tcl.call('set', 'a', '\u20ac\0') # non-ASCII + self.assertEqual(tcl.getvar('a'), '\u20ac\x00') + self.assertEqual(tcl.call('set', 'a'), '\u20ac\x00') + self.assertEqual(tcl.eval('set a'), '\u20ac\x00') + def testCallException(self): tcl = self.interp self.assertRaises(TclError,tcl.call,'set','a') @@ -98,6 +110,18 @@ def testSetVar(self): tcl.setvar('a','1') self.assertEqual(tcl.eval('set a'),'1') + def test_setvar_passing_null(self): + tcl = self.interp + tcl.setvar('a', 'a\0b') # ASCII-only + self.assertEqual(tcl.getvar('a'), 'a\x00b') + self.assertEqual(tcl.call('set', 'a'), 'a\x00b') + self.assertEqual(tcl.eval('set a'), 'a\x00b') + + tcl.setvar('a', '\u20ac\0') # non-ASCII + self.assertEqual(tcl.getvar('a'), '\u20ac\x00') + self.assertEqual(tcl.call('set', 'a'), '\u20ac\x00') + self.assertEqual(tcl.eval('set a'), '\u20ac\x00') + def testSetVarArray(self): tcl = self.interp tcl.setvar('a(1)','1') diff --git a/Lib/test/test_tkinter/test_misc.py b/Lib/test/test_tkinter/test_misc.py index d9ea642881a179..b0b9ed60040443 100644 --- a/Lib/test/test_tkinter/test_misc.py +++ b/Lib/test/test_tkinter/test_misc.py @@ -476,6 +476,15 @@ def test_info_patchlevel(self): self.assertEqual(vi.micro, 0) self.assertTrue(str(vi).startswith(f'{vi.major}.{vi.minor}')) + def test_embedded_null(self): + widget = tkinter.Entry(self.root) + widget.insert(0, 'abc\0def') # ASCII-only + widget.selection_range(0, 'end') + self.assertEqual(widget.selection_get(), 'abc\x00def') + widget.insert(0, '\u20ac\0') # non-ASCII + widget.selection_range(0, 'end') + self.assertEqual(widget.selection_get(), '\u20ac\0abc\x00def') + class WmTest(AbstractTkTest, unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst b/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst new file mode 100644 index 00000000000000..d518265a7fe55a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-06-23-17-50-40.gh-issue-119614.vwPGLB.rst @@ -0,0 +1,2 @@ +Fix truncation of strings with embedded null characters in some internal +operations in :mod:`tkinter`. diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index 8a5347be7de6ad..1542d3af42f755 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -512,7 +512,7 @@ unicodeFromTclObj(TkappObject *tkapp, Tcl_Obj *value) else Py_UNREACHABLE(); } -#endif +#endif /* USE_TCL_UNICODE */ const char *s = Tcl_GetStringFromObj(value, &len); return unicodeFromTclStringAndSize(s, len); } @@ -1018,7 +1018,9 @@ AsObj(PyObject *value) PyErr_SetString(PyExc_OverflowError, "string is too long"); return NULL; } - if (PyUnicode_IS_ASCII(value)) { + if (PyUnicode_IS_ASCII(value) && + strlen(PyUnicode_DATA(value)) == (size_t)PyUnicode_GET_LENGTH(value)) + { return Tcl_NewStringObj((const char *)PyUnicode_DATA(value), (int)size); } @@ -1033,9 +1035,6 @@ AsObj(PyObject *value) "surrogatepass", NATIVE_BYTEORDER); else Py_UNREACHABLE(); -#else - encoded = _PyUnicode_AsUTF8String(value, "surrogateescape"); -#endif if (!encoded) { return NULL; } @@ -1045,12 +1044,39 @@ AsObj(PyObject *value) PyErr_SetString(PyExc_OverflowError, "string is too long"); return NULL; } -#if USE_TCL_UNICODE result = Tcl_NewUnicodeObj((const Tcl_UniChar *)PyBytes_AS_STRING(encoded), (int)(size / sizeof(Tcl_UniChar))); #else + encoded = _PyUnicode_AsUTF8String(value, "surrogateescape"); + if (!encoded) { + return NULL; + } + size = PyBytes_GET_SIZE(encoded); + if (strlen(PyBytes_AS_STRING(encoded)) != (size_t)size) { + /* The string contains embedded null characters. + * Tcl needs a null character to be represented as \xc0\x80 in + * the Modified UTF-8 encoding. Otherwise the string can be + * truncated in some internal operations. + * + * NOTE: stringlib_replace() could be used here, but optimizing + * this obscure case isn't worth it unless stringlib_replace() + * was already exposed in the C API for other reasons. */ + Py_SETREF(encoded, + PyObject_CallMethod(encoded, "replace", "y#y#", + "\0", (Py_ssize_t)1, + "\xc0\x80", (Py_ssize_t)2)); + if (!encoded) { + return NULL; + } + size = PyBytes_GET_SIZE(encoded); + } + if (size > INT_MAX) { + Py_DECREF(encoded); + PyErr_SetString(PyExc_OverflowError, "string is too long"); + return NULL; + } result = Tcl_NewStringObj(PyBytes_AS_STRING(encoded), (int)size); -#endif +#endif /* USE_TCL_UNICODE */ Py_DECREF(encoded); return result; }

1 0

[3.13] gh-112169: Documented getaddrinfo/getnameinfo default loop executor usage and implications. (GH-112191) (#120935)
by kumaraditya303 24 Jun '24

24 Jun '24

https://github.com/python/cpython/commit/206028dba986f982a940377ab1cb8b8276… commit: 206028dba986f982a940377ab1cb8b8276301b82 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: kumaraditya303 <kumaraditya(a)python.org> date: 2024-06-24T08:55:28Z summary: [3.13] gh-112169: Documented getaddrinfo/getnameinfo default loop executor usage and implications. (GH-112191) (#120935) gh-112169: Documented getaddrinfo/getnameinfo default loop executor usage and implications. (GH-112191) (cherry picked from commit fc297b4ba4c61febeb2d8f5d718f2955c6bbea0a) Co-authored-by: Alek Kowalczyk <alek.kowalczyk(a)gmail.com> Co-authored-by: Guido van Rossum <gvanrossum(a)gmail.com> Co-authored-by: Kumar Aditya <kumaraditya(a)python.org> Co-authored-by: Carol Willing <carolcode(a)willingconsulting.com> files: M Doc/library/asyncio-eventloop.rst diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 374e789e91e790..1d79f78e8e1b67 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -1155,6 +1155,14 @@ DNS Asynchronous version of :meth:`socket.getnameinfo`. +.. note:: + Both *getaddrinfo* and *getnameinfo* internally utilize their synchronous + versions through the loop's default thread pool executor. + When this executor is saturated, these methods may experience delays, + which higher-level networking libraries may report as increased timeouts. + To mitigate this, consider using a custom executor for other user tasks, + or setting a default executor with a larger number of workers. + .. versionchanged:: 3.7 Both *getaddrinfo* and *getnameinfo* methods were always documented to return a coroutine, but prior to Python 3.7 they were, in fact,

1 0