Mailman 3 November 2024 - Python-checkins

[3.13] gh-109746: Fix race condition in test_start_new_thread_failed (GH-127299) (GH-127323)
by serhiy-storchaka Nov. 30, 2024

Nov. 30, 2024

https://github.com/python/cpython/commit/82d199d307e0f5ab65a3035040af26a370… commit: 82d199d307e0f5ab65a3035040af26a370d6e2d4 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2024-11-30T09:19:13Z summary: [3.13] gh-109746: Fix race condition in test_start_new_thread_failed (GH-127299) (GH-127323) (cherry picked from commit 83926d3b4c7847394b5e2531e9566d7fc9fbea0f) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: M Lib/test/test_threading.py diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index 7ee0b2964bdb53..c13d1bd0f81efb 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -1196,11 +1196,11 @@ def f(): except RuntimeError: print('ok') else: - print('skip') + print('!skip!') """ _, out, err = assert_python_ok("-u", "-c", code) out = out.strip() - if out == b'skip': + if b'!skip!' in out: self.skipTest('RLIMIT_NPROC had no effect; probably superuser') self.assertEqual(out, b'ok') self.assertEqual(err, b'')

1 0

[3.13] gh-124008: Fix calculation of the number of written bytes for the Windows console (GH-124059) (GH-127325)
by serhiy-storchaka Nov. 30, 2024

Nov. 30, 2024

https://github.com/python/cpython/commit/d2d1055129df2184e11947c25380da0419… commit: d2d1055129df2184e11947c25380da041959c347 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2024-11-30T09:16:45Z summary: [3.13] gh-124008: Fix calculation of the number of written bytes for the Windows console (GH-124059) (GH-127325) Since MultiByteToWideChar()/WideCharToMultiByte() is not reversible if the data contains invalid UTF-8 sequences, use binary search to calculate the number of written bytes from the number of written characters. Also fix writing incomplete UTF-8 sequences. Also fix handling of memory allocation failures. (cherry picked from commit 3cf83d91a5baf3600dd60f7aaaf4fb6d73c4b8a9) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: A Misc/NEWS.d/next/Library/2024-09-13-18-24-27.gh-issue-124008.XaiPQx.rst M Lib/test/test_winconsoleio.py M Modules/_io/winconsoleio.c diff --git a/Lib/test/test_winconsoleio.py b/Lib/test/test_winconsoleio.py index a10d63dfdc9753..d9076e77c158a2 100644 --- a/Lib/test/test_winconsoleio.py +++ b/Lib/test/test_winconsoleio.py @@ -142,6 +142,29 @@ def test_write_empty_data(self): with ConIO('CONOUT$', 'w') as f: self.assertEqual(f.write(b''), 0) + @requires_resource('console') + def test_write(self): + testcases = [] + with ConIO('CONOUT$', 'w') as f: + for a in [ + b'', + b'abc', + b'\xc2\xa7\xe2\x98\x83\xf0\x9f\x90\x8d', + b'\xff'*10, + ]: + for b in b'\xc2\xa7', b'\xe2\x98\x83', b'\xf0\x9f\x90\x8d': + testcases.append(a + b) + for i in range(1, len(b)): + data = a + b[:i] + testcases.append(data + b'z') + testcases.append(data + b'\xff') + # incomplete multibyte sequence + with self.subTest(data=data): + self.assertEqual(f.write(data), len(a)) + for data in testcases: + with self.subTest(data=data): + self.assertEqual(f.write(data), len(data)) + def assertStdinRoundTrip(self, text): stdin = open('CONIN$', 'r') old_stdin = sys.stdin diff --git a/Misc/NEWS.d/next/Library/2024-09-13-18-24-27.gh-issue-124008.XaiPQx.rst b/Misc/NEWS.d/next/Library/2024-09-13-18-24-27.gh-issue-124008.XaiPQx.rst new file mode 100644 index 00000000000000..cd6dd9a7a97e90 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-09-13-18-24-27.gh-issue-124008.XaiPQx.rst @@ -0,0 +1,2 @@ +Fix possible crash (in debug build), incorrect output or returning incorrect +value from raw binary ``write()`` when writing to console on Windows. diff --git a/Modules/_io/winconsoleio.c b/Modules/_io/winconsoleio.c index ec5c298066a587..75c61d7ad3c453 100644 --- a/Modules/_io/winconsoleio.c +++ b/Modules/_io/winconsoleio.c @@ -135,19 +135,67 @@ char _PyIO_get_console_type(PyObject *path_or_fd) { } static DWORD -_find_last_utf8_boundary(const char *buf, DWORD len) +_find_last_utf8_boundary(const unsigned char *buf, DWORD len) { - /* This function never returns 0, returns the original len instead */ - DWORD count = 1; - if (len == 0 || (buf[len - 1] & 0x80) == 0) { - return len; - } - for (;; count++) { - if (count > 3 || count >= len) { + for (DWORD count = 1; count < 4 && count <= len; count++) { + unsigned char c = buf[len - count]; + if (c < 0x80) { + /* No starting byte found. */ return len; } - if ((buf[len - count] & 0xc0) != 0x80) { - return len - count; + if (c >= 0xc0) { + if (c < 0xe0 /* 2-bytes sequence */ ? count < 2 : + c < 0xf0 /* 3-bytes sequence */ ? count < 3 : + c < 0xf8 /* 4-bytes sequence */) + { + /* Incomplete multibyte sequence. */ + return len - count; + } + /* Either complete or invalid sequence. */ + return len; + } + } + /* Either complete 4-bytes sequence or invalid sequence. */ + return len; +} + +/* Find the number of UTF-8 bytes that corresponds to the specified number of + * wchars. + * I.e. find x <= len so that MultiByteToWideChar(CP_UTF8, 0, s, x, NULL, 0) == n. + * + * WideCharToMultiByte() cannot be used for this, because the UTF-8 -> wchar + * conversion is not reversible (invalid UTF-8 byte produces \ufffd which + * will be converted back to 3-bytes UTF-8 sequence \xef\xbf\xbd). + * So we need to use binary search. + */ +static DWORD +_wchar_to_utf8_count(const unsigned char *s, DWORD len, DWORD n) +{ + DWORD start = 0; + while (1) { + DWORD mid = 0; + for (DWORD i = len / 2; i <= len; i++) { + mid = _find_last_utf8_boundary(s, i); + if (mid != 0) { + break; + } + /* The middle could split the first multibytes sequence. */ + } + if (mid == len) { + return start + len; + } + if (mid == 0) { + mid = len > 1 ? len - 1 : 1; + } + DWORD wlen = MultiByteToWideChar(CP_UTF8, 0, s, mid, NULL, 0); + if (wlen <= n) { + s += mid; + start += mid; + len -= mid; + n -= wlen; + } + else { + len = mid; } } } @@ -563,8 +611,10 @@ read_console_w(HANDLE handle, DWORD maxlen, DWORD *readlen) { int err = 0, sig = 0; wchar_t *buf = (wchar_t*)PyMem_Malloc(maxlen * sizeof(wchar_t)); - if (!buf) + if (!buf) { + PyErr_NoMemory(); goto error; + } *readlen = 0; @@ -622,6 +672,7 @@ read_console_w(HANDLE handle, DWORD maxlen, DWORD *readlen) { Py_UNBLOCK_THREADS if (!newbuf) { sig = -1; + PyErr_NoMemory(); break; } buf = newbuf; @@ -645,8 +696,10 @@ read_console_w(HANDLE handle, DWORD maxlen, DWORD *readlen) { if (*readlen > 0 && buf[0] == L'\x1a') { PyMem_Free(buf); buf = (wchar_t *)PyMem_Malloc(sizeof(wchar_t)); - if (!buf) + if (!buf) { + PyErr_NoMemory(); goto error; + } buf[0] = L'\0'; *readlen = 0; } @@ -824,8 +877,10 @@ _io__WindowsConsoleIO_readall_impl(winconsoleio *self) bufsize = BUFSIZ; buf = (wchar_t*)PyMem_Malloc((bufsize + 1) * sizeof(wchar_t)); - if (buf == NULL) + if (buf == NULL) { + PyErr_NoMemory(); return NULL; + } while (1) { wchar_t *subbuf; @@ -847,6 +902,7 @@ _io__WindowsConsoleIO_readall_impl(winconsoleio *self) (bufsize + 1) * sizeof(wchar_t)); if (tmp == NULL) { PyMem_Free(buf); + PyErr_NoMemory(); return NULL; } buf = tmp; @@ -1022,43 +1078,49 @@ _io__WindowsConsoleIO_write_impl(winconsoleio *self, PyTypeObject *cls, len = (DWORD)b->len; Py_BEGIN_ALLOW_THREADS - wlen = MultiByteToWideChar(CP_UTF8, 0, b->buf, len, NULL, 0); - /* issue11395 there is an unspecified upper bound on how many bytes can be written at once. We cap at 32k - the caller will have to handle partial writes. Since we don't know how many input bytes are being ignored, we have to reduce and recalculate. */ - while (wlen > 32766 / sizeof(wchar_t)) { - len /= 2; + const DWORD max_wlen = 32766 / sizeof(wchar_t); + /* UTF-8 to wchar ratio is at most 3:1. */ + len = Py_MIN(len, max_wlen * 3); + while (1) { /* Fix for github issues gh-110913 and gh-82052. */ len = _find_last_utf8_boundary(b->buf, len); wlen = MultiByteToWideChar(CP_UTF8, 0, b->buf, len, NULL, 0); + if (wlen <= max_wlen) { + break; + } + len /= 2; } Py_END_ALLOW_THREADS - if (!wlen) - return PyErr_SetFromWindowsErr(0); + if (!wlen) { + return PyLong_FromLong(0); + } wbuf = (wchar_t*)PyMem_Malloc(wlen * sizeof(wchar_t)); + if (!wbuf) { + PyErr_NoMemory(); + return NULL; + } Py_BEGIN_ALLOW_THREADS wlen = MultiByteToWideChar(CP_UTF8, 0, b->buf, len, wbuf, wlen); if (wlen) { res = WriteConsoleW(handle, wbuf, wlen, &n, NULL); +#ifdef Py_DEBUG + if (res) { +#else if (res && n < wlen) { +#endif /* Wrote fewer characters than expected, which means our * len value may be wrong. So recalculate it from the - * characters that were written. As this could potentially - * result in a different value, we also validate that value. + * characters that were written. */ - len = WideCharToMultiByte(CP_UTF8, 0, wbuf, n, - NULL, 0, NULL, NULL); - if (len) { - wlen = MultiByteToWideChar(CP_UTF8, 0, b->buf, len, - NULL, 0); - assert(wlen == len); - } + len = _wchar_to_utf8_count(b->buf, len, n); } } else res = 0;

1 0

[3.13] gh-123967: Fix faulthandler for trampoline frames (GH-127329) (#127362)
by vstinner Nov. 30, 2024

Nov. 30, 2024

https://github.com/python/cpython/commit/089d70614e6dcb8c27a537a03ac3e70196… commit: 089d70614e6dcb8c27a537a03ac3e701960ba629 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: vstinner <vstinner(a)python.org> date: 2024-11-30T09:14:29Z summary: [3.13] gh-123967: Fix faulthandler for trampoline frames (GH-127329) (#127362) gh-123967: Fix faulthandler for trampoline frames (GH-127329) If the top-most frame is a trampoline frame, skip it. (cherry picked from commit 58e334e1431b2ed6b70ee42501ea73e08084e769) Co-authored-by: Victor Stinner <vstinner(a)python.org> files: A Misc/NEWS.d/next/Library/2024-11-27-14-06-35.gh-issue-123967.wxUmnW.rst M Python/traceback.c diff --git a/Misc/NEWS.d/next/Library/2024-11-27-14-06-35.gh-issue-123967.wxUmnW.rst b/Misc/NEWS.d/next/Library/2024-11-27-14-06-35.gh-issue-123967.wxUmnW.rst new file mode 100644 index 00000000000000..788fe0c78ef257 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-27-14-06-35.gh-issue-123967.wxUmnW.rst @@ -0,0 +1,2 @@ +Fix faulthandler for trampoline frames. If the top-most frame is a +trampoline frame, skip it. Patch by Victor Stinner. diff --git a/Python/traceback.c b/Python/traceback.c index 47b77c9108dd9a..e819909b6045c3 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -890,6 +890,8 @@ _Py_DumpASCII(int fd, PyObject *text) static void dump_frame(int fd, _PyInterpreterFrame *frame) { + assert(frame->owner != FRAME_OWNED_BY_CSTACK); + PyCodeObject *code =_PyFrame_GetCode(frame); PUTS(fd, " File "); if (code->co_filename != NULL @@ -963,6 +965,17 @@ dump_traceback(int fd, PyThreadState *tstate, int write_header) unsigned int depth = 0; while (1) { + if (frame->owner == FRAME_OWNED_BY_CSTACK) { + /* Trampoline frame */ + frame = frame->previous; + if (frame == NULL) { + break; + } + + /* Can't have more than one shim frame in a row */ + assert(frame->owner != FRAME_OWNED_BY_CSTACK); + } + if (MAX_FRAME_DEPTH <= depth) { if (MAX_FRAME_DEPTH < depth) { PUTS(fd, "plus "); @@ -971,20 +984,12 @@ dump_traceback(int fd, PyThreadState *tstate, int write_header) } break; } + dump_frame(fd, frame); frame = frame->previous; if (frame == NULL) { break; } - if (frame->owner == FRAME_OWNED_BY_CSTACK) { - /* Trampoline frame */ - frame = frame->previous; - } - if (frame == NULL) { - break; - } - /* Can't have more than one shim frame in a row */ - assert(frame->owner != FRAME_OWNED_BY_CSTACK); depth++; } }

1 0

[3.13] gh-127217: Fix pathname2url() for paths starting with multiple slashes on Posix (GH-127218) (GH-127230)
by serhiy-storchaka Nov. 30, 2024

Nov. 30, 2024

https://github.com/python/cpython/commit/c711deb803d154ef3155406eb33e711136… commit: c711deb803d154ef3155406eb33e71113684931c branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2024-11-30T09:11:24Z summary: [3.13] gh-127217: Fix pathname2url() for paths starting with multiple slashes on Posix (GH-127218) (GH-127230) (cherry picked from commit 97b2ceaaaf88a73a45254912a0e972412879ccbf) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: A Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst M Lib/test/test_urllib.py M Lib/urllib/request.py diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py index 5542d4d2b26ca4..6539496370116a 100644 --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -1556,6 +1556,9 @@ def test_pathname2url_posix(self): fn = urllib.request.pathname2url self.assertEqual(fn('/'), '/') self.assertEqual(fn('/a/b.c'), '/a/b.c') + self.assertEqual(fn('//a/b.c'), '////a/b.c') + self.assertEqual(fn('///a/b.c'), '/////a/b.c') + self.assertEqual(fn('////a/b.c'), '//////a/b.c') self.assertEqual(fn('/a/b%#c'), '/a/b%25%23c') @unittest.skipUnless(os_helper.FS_NONASCII, 'need os_helper.FS_NONASCII') diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index de31b0c853540b..b4882f4129e3a6 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -1670,6 +1670,10 @@ def url2pathname(pathname): def pathname2url(pathname): """OS-specific conversion from a file system path to a relative URL of the 'file' scheme; not recommended for general use.""" + if pathname[:2] == '//': + # Add explicitly empty authority to avoid interpreting the path + # as authority. + pathname = '//' + pathname encoding = sys.getfilesystemencoding() errors = sys.getfilesystemencodeerrors() return quote(pathname, encoding=encoding, errors=errors) diff --git a/Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst b/Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst new file mode 100644 index 00000000000000..3139e33302f378 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst @@ -0,0 +1,2 @@ +Fix :func:`urllib.request.pathname2url` for paths starting with multiple +slashes on Posix.

1 0

[3.12] gh-88110: Clear concurrent.futures.thread._threads_queues after fork to avoid joining parent process' threads (GH-126098) (GH-127164)
by serhiy-storchaka Nov. 30, 2024

Nov. 30, 2024

https://github.com/python/cpython/commit/7c7ed49b518f986806edf2f3d67198cb51… commit: 7c7ed49b518f986806edf2f3d67198cb516b7a98 branch: 3.12 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2024-11-30T09:05:46Z summary: [3.12] gh-88110: Clear concurrent.futures.thread._threads_queues after fork to avoid joining parent process' threads (GH-126098) (GH-127164) Threads are gone after fork, so clear the queues too. Otherwise the child process (here created via multiprocessing.Process) crashes on interpreter exit. (cherry picked from commit 1848ce61f349533ae5892a8c24c2e0e3c364fc8a) Co-authored-by: Andrei Bodrov <Drino(a)users.noreply.github.com> Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: A Misc/NEWS.d/next/Library/2023-02-15-23-54-42.gh-issue-88110.KU6erv.rst M Lib/concurrent/futures/thread.py M Lib/test/test_concurrent_futures/test_thread_pool.py diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py index 3b3a36a5093336..61dbff8a485bcd 100644 --- a/Lib/concurrent/futures/thread.py +++ b/Lib/concurrent/futures/thread.py @@ -41,6 +41,7 @@ def _python_exit(): os.register_at_fork(before=_global_shutdown_lock.acquire, after_in_child=_global_shutdown_lock._at_fork_reinit, after_in_parent=_global_shutdown_lock.release) + os.register_at_fork(after_in_child=_threads_queues.clear) class _WorkItem: diff --git a/Lib/test/test_concurrent_futures/test_thread_pool.py b/Lib/test/test_concurrent_futures/test_thread_pool.py index 812f989d8f3ad2..6e4a4b7caff481 100644 --- a/Lib/test/test_concurrent_futures/test_thread_pool.py +++ b/Lib/test/test_concurrent_futures/test_thread_pool.py @@ -64,6 +64,25 @@ def submit(pool): with futures.ProcessPoolExecutor(1, mp_context=mp.get_context('fork')) as workers: workers.submit(tuple) + @support.requires_fork() + @unittest.skipUnless(hasattr(os, 'register_at_fork'), 'need os.register_at_fork') + def test_process_fork_from_a_threadpool(self): + # bpo-43944: clear concurrent.futures.thread._threads_queues after fork, + # otherwise child process will try to join parent thread + def fork_process_and_return_exitcode(): + # Ignore the warning about fork with threads. + with self.assertWarnsRegex(DeprecationWarning, + r"use of fork may lead to deadlocks in the child"): + p = mp.get_context('fork').Process(target=lambda: 1) + p.start() + p.join() + return p.exitcode + + with futures.ThreadPoolExecutor(1) as pool: + process_exitcode = pool.submit(fork_process_and_return_exitcode).result() + + self.assertEqual(process_exitcode, 0) + def test_executor_map_current_future_cancel(self): stop_event = threading.Event() log = [] diff --git a/Misc/NEWS.d/next/Library/2023-02-15-23-54-42.gh-issue-88110.KU6erv.rst b/Misc/NEWS.d/next/Library/2023-02-15-23-54-42.gh-issue-88110.KU6erv.rst new file mode 100644 index 00000000000000..42a83edc3ba68d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-02-15-23-54-42.gh-issue-88110.KU6erv.rst @@ -0,0 +1,2 @@ +Fixed :class:`multiprocessing.Process` reporting a ``.exitcode`` of 1 even on success when +using the ``"fork"`` start method while using a :class:`concurrent.futures.ThreadPoolExecutor`.

1 0

[3.13] gh-127303: Add docs for token.EXACT_TOKEN_TYPES (GH-127304) (#127390)
by terryjreedy Nov. 30, 2024

Nov. 30, 2024

https://github.com/python/cpython/commit/4cba0e66c29b46afbb1eee1d0428f5a2f5… commit: 4cba0e66c29b46afbb1eee1d0428f5a2f5b891bb branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: terryjreedy <tjreedy(a)udel.edu> date: 2024-11-30T08:13:13Z summary: [3.13] gh-127303: Add docs for token.EXACT_TOKEN_TYPES (GH-127304) (#127390) gh-127303: Add docs for token.EXACT_TOKEN_TYPES (GH-127304) --------- (cherry picked from commit dd3a87d2a8f8750978359a99de2c5cb2168351d1) Co-authored-by: Илья Любавский <100635212+lubaskinc0de(a)users.noreply.github.com> Co-authored-by: Terry Jan Reedy <tjreedy(a)udel.edu> Co-authored-by: Tomas R. <tomas.roun8(a)gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2024-11-27-16-06-10.gh-issue-127303.asqkgh.rst M Doc/library/token.rst M Lib/token.py M Misc/ACKS M Tools/build/generate_token.py diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 0cc9dddd91ed6b..40982f32b4beee 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -79,6 +79,13 @@ the :mod:`tokenize` module. ``type_comments=True``. +.. data:: EXACT_TOKEN_TYPES + + A dictionary mapping the string representation of a token to its numeric code. + + .. versionadded:: 3.8 + + .. versionchanged:: 3.5 Added :data:`!AWAIT` and :data:`!ASYNC` tokens. diff --git a/Lib/token.py b/Lib/token.py index b620317106e173..54d7cdccadc79a 100644 --- a/Lib/token.py +++ b/Lib/token.py @@ -1,7 +1,8 @@ """Token constants.""" # Auto-generated by Tools/build/generate_token.py -__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF'] +__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF', + 'EXACT_TOKEN_TYPES'] ENDMARKER = 0 NAME = 1 diff --git a/Misc/ACKS b/Misc/ACKS index d3bf8450fcbb4f..bed3e028d0a18d 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1149,6 +1149,7 @@ Mark Lutz Taras Lyapun Jim Lynch Mikael Lyngvig +Ilya Lyubavski Jeff MacDonald John Machin Andrew I MacIntyre diff --git a/Misc/NEWS.d/next/Library/2024-11-27-16-06-10.gh-issue-127303.asqkgh.rst b/Misc/NEWS.d/next/Library/2024-11-27-16-06-10.gh-issue-127303.asqkgh.rst new file mode 100644 index 00000000000000..58ebf5d0abe141 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-27-16-06-10.gh-issue-127303.asqkgh.rst @@ -0,0 +1 @@ +Publicly expose :data:`~token.EXACT_TOKEN_TYPES` in :attr:`!token.__all__`. diff --git a/Tools/build/generate_token.py b/Tools/build/generate_token.py index 16c38841e44a4d..d32747f19945d8 100755 --- a/Tools/build/generate_token.py +++ b/Tools/build/generate_token.py @@ -226,7 +226,8 @@ def make_rst(infile, outfile='Doc/library/token-list.inc'): # {AUTO_GENERATED_BY_SCRIPT} ''' token_py_template += ''' -__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF'] +__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF', + 'EXACT_TOKEN_TYPES'] %s N_TOKENS = %d

1 0

gh-119786: improve internal docs on `co_linetable` (#123198)
by iritkatriel Nov. 30, 2024

Nov. 30, 2024

https://github.com/python/cpython/commit/49f15d8667e4755876698a8daa13bab6ac… commit: 49f15d8667e4755876698a8daa13bab6acee5fa1 branch: main author: Bénédikt Tran <10796600+picnixz(a)users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel(a)users.noreply.github.com> date: 2024-11-30T00:25:55Z summary: gh-119786: improve internal docs on `co_linetable` (#123198) files: M InternalDocs/code_objects.md diff --git a/InternalDocs/code_objects.md b/InternalDocs/code_objects.md index bee4a9d0a08915..d4e28c6b238b48 100644 --- a/InternalDocs/code_objects.md +++ b/InternalDocs/code_objects.md @@ -1,4 +1,3 @@ - # Code objects A `CodeObject` is a builtin Python type that represents a compiled executable, @@ -43,7 +42,7 @@ so a compact format is very important. Note that traceback objects don't store all this information -- they store the start line number, for backward compatibility, and the "last instruction" value. The rest can be computed from the last instruction (`tb_lasti`) with the help of the -locations table. For Python code, there is a convenience method +locations table. For Python code, there is a convenience method (`codeobject.co_positions`)[https://docs.python.org/dev/reference/datamodel.html#codeobject.co_positions] which returns an iterator of `({line}, {endline}, {column}, {endcolumn})` tuples, one per instruction. @@ -75,9 +74,11 @@ returned by the `co_positions()` iterator. > See [`Objects/lnotab_notes.txt`](../Objects/lnotab_notes.txt) for more details. `co_linetable` consists of a sequence of location entries. -Each entry starts with a byte with the most significant bit set, followed by zero or more bytes with the most significant bit unset. +Each entry starts with a byte with the most significant bit set, followed by +zero or more bytes with the most significant bit unset. Each entry contains the following information: + * The number of code units covered by this entry (length) * The start line * The end line @@ -86,54 +87,88 @@ Each entry contains the following information: The first byte has the following format: -Bit 7 | Bits 3-6 | Bits 0-2 - ---- | ---- | ---- - 1 | Code | Length (in code units) - 1 +| Bit 7 | Bits 3-6 | Bits 0-2 | +|-------|----------|----------------------------| +| 1 | Code | Length (in code units) - 1 | The codes are enumerated in the `_PyCodeLocationInfoKind` enum. -## Variable-length integer encodings +### Variable-length integer encodings -Integers are often encoded using a variable-length integer encoding +Integers are often encoded using a variable length integer encoding -### Unsigned integers (`varint`) +#### Unsigned integers (`varint`) Unsigned integers are encoded in 6-bit chunks, least significant first. Each chunk but the last has bit 6 set. For example: * 63 is encoded as `0x3f` -* 200 is encoded as `0x48`, `0x03` +* 200 is encoded as `0x48`, `0x03` since ``200 = (0x03 << 6) | 0x48``. + +The following helper can be used to convert an integer into a `varint`: + +```py +def encode_varint(s): + ret = [] + while s >= 64: + ret.append(((s & 0x3F) | 0x40) & 0x3F) + s >>= 6 + ret.append(s & 0x3F) + return bytes(ret) +``` + +To convert a `varint` into an unsigned integer: + +```py +def decode_varint(chunks): + ret = 0 + for chunk in reversed(chunks): + ret = (ret << 6) | chunk + return ret +``` -### Signed integers (`svarint`) +#### Signed integers (`svarint`) Signed integers are encoded by converting them to unsigned integers, using the following function: -```Python -def convert(s): + +```py +def svarint_to_varint(s): if s < 0: - return ((-s)<<1) | 1 + return ((-s) << 1) | 1 else: - return (s<<1) + return s << 1 +``` + +To convert a `varint` into a signed integer: + +```py +def varint_to_svarint(uval): + return -(uval >> 1) if uval & 1 else (uval >> 1) ``` -*Location entries* +### Location entries The meaning of the codes and the following bytes are as follows: -Code | Meaning | Start line | End line | Start column | End column - ---- | ---- | ---- | ---- | ---- | ---- - 0-9 | Short form | Δ 0 | Δ 0 | See below | See below - 10-12 | One line form | Δ (code - 10) | Δ 0 | unsigned byte | unsigned byte - 13 | No column info | Δ svarint | Δ 0 | None | None - 14 | Long form | Δ svarint | Δ varint | varint | varint - 15 | No location | None | None | None | None +| Code | Meaning | Start line | End line | Start column | End column | +|-------|----------------|---------------|----------|---------------|---------------| +| 0-9 | Short form | Δ 0 | Δ 0 | See below | See below | +| 10-12 | One line form | Δ (code - 10) | Δ 0 | unsigned byte | unsigned byte | +| 13 | No column info | Δ svarint | Δ 0 | None | None | +| 14 | Long form | Δ svarint | Δ varint | varint | varint | +| 15 | No location | None | None | None | None | The Δ means the value is encoded as a delta from another value: + * Start line: Delta from the previous start line, or `co_firstlineno` for the first entry. -* End line: Delta from the start line +* End line: Delta from the start line. + +### The short forms -*The short forms* +Codes 0-9 are the short forms. The short form consists of two bytes, +the second byte holding additional column information. The code is the +start column divided by 8 (and rounded down). -Codes 0-9 are the short forms. The short form consists of two bytes, the second byte holding additional column information. The code is the start column divided by 8 (and rounded down). * Start column: `(code*8) + ((second_byte>>4)&7)` * End column: `start_column + (second_byte&15)`

1 0

GH-127381: pathlib ABCs: remove `PathBase.lstat()` (#127382)
by barneygale Nov. 29, 2024

Nov. 29, 2024

https://github.com/python/cpython/commit/38264a060a8178d58046e90e9beb8220e3… commit: 38264a060a8178d58046e90e9beb8220e3c22046 branch: main author: Barney Gale <barney.gale(a)gmail.com> committer: barneygale <barney.gale(a)gmail.com> date: 2024-11-29T21:03:39Z summary: GH-127381: pathlib ABCs: remove `PathBase.lstat()` (#127382) Remove the `PathBase.lstat()` method, which is a trivial variation of `stat()`. No user-facing changes because the pathlib ABCs are still private. files: M Lib/pathlib/_abc.py M Lib/pathlib/_local.py M Lib/test/test_pathlib/test_pathlib.py M Lib/test/test_pathlib/test_pathlib_abc.py diff --git a/Lib/pathlib/_abc.py b/Lib/pathlib/_abc.py index 2c243d470d4eda..697f32985ff652 100644 --- a/Lib/pathlib/_abc.py +++ b/Lib/pathlib/_abc.py @@ -438,14 +438,6 @@ def stat(self, *, follow_symlinks=True): """ raise UnsupportedOperation(self._unsupported_msg('stat()')) - def lstat(self): - """ - Like stat(), except if the path points to a symlink, the symlink's - status information is returned, rather than its target's. - """ - return self.stat(follow_symlinks=False) - - # Convenience functions for querying the stat results def exists(self, *, follow_symlinks=True): @@ -505,7 +497,7 @@ def is_symlink(self): Whether this path is a symbolic link. """ try: - return S_ISLNK(self.lstat().st_mode) + return S_ISLNK(self.stat(follow_symlinks=False).st_mode) except (OSError, ValueError): return False @@ -789,7 +781,7 @@ def raise_error(*args): def lstat(path_str): path = self.with_segments(path_str) path._resolving = True - return path.lstat() + return path.stat(follow_symlinks=False) def readlink(path_str): path = self.with_segments(path_str) diff --git a/Lib/pathlib/_local.py b/Lib/pathlib/_local.py index b27f456d375225..25c1e3f44ea7e8 100644 --- a/Lib/pathlib/_local.py +++ b/Lib/pathlib/_local.py @@ -542,6 +542,13 @@ def stat(self, *, follow_symlinks=True): """ return os.stat(self, follow_symlinks=follow_symlinks) + def lstat(self): + """ + Like stat(), except if the path points to a symlink, the symlink's + status information is returned, rather than its target's. + """ + return os.lstat(self) + def exists(self, *, follow_symlinks=True): """ Whether this path exists. diff --git a/Lib/test/test_pathlib/test_pathlib.py b/Lib/test/test_pathlib/test_pathlib.py index 6a994f890da616..2c48eeeda145d0 100644 --- a/Lib/test/test_pathlib/test_pathlib.py +++ b/Lib/test/test_pathlib/test_pathlib.py @@ -546,12 +546,9 @@ def tempdir(self): self.addCleanup(os_helper.rmtree, d) return d - def test_matches_pathbase_api(self): - our_names = {name for name in dir(self.cls) if name[0] != '_'} - our_names.remove('is_reserved') # only present in PurePath + def test_matches_pathbase_docstrings(self): path_names = {name for name in dir(pathlib._abc.PathBase) if name[0] != '_'} - self.assertEqual(our_names, path_names) - for attr_name in our_names: + for attr_name in path_names: if attr_name == 'parser': # On Windows, Path.parser is ntpath, but PathBase.parser is # posixpath, and so their docstrings differ. @@ -1357,6 +1354,17 @@ def test_symlink_to_unsupported(self): with self.assertRaises(pathlib.UnsupportedOperation): q.symlink_to(p) + @needs_symlinks + def test_lstat(self): + p = self.cls(self.base)/ 'linkA' + st = p.stat() + self.assertNotEqual(st, p.lstat()) + + def test_lstat_nosymlink(self): + p = self.cls(self.base) / 'fileA' + st = p.stat() + self.assertEqual(st, p.lstat()) + def test_is_junction(self): P = self.cls(self.base) diff --git a/Lib/test/test_pathlib/test_pathlib_abc.py b/Lib/test/test_pathlib/test_pathlib_abc.py index aaa30a17f2af14..7ca35e3dc7ee00 100644 --- a/Lib/test/test_pathlib/test_pathlib_abc.py +++ b/Lib/test/test_pathlib/test_pathlib_abc.py @@ -1351,7 +1351,6 @@ def test_unsupported_operation(self): p = self.cls('') e = UnsupportedOperation self.assertRaises(e, p.stat) - self.assertRaises(e, p.lstat) self.assertRaises(e, p.exists) self.assertRaises(e, p.samefile, 'foo') self.assertRaises(e, p.is_dir) @@ -2671,17 +2670,6 @@ def test_stat_no_follow_symlinks_nosymlink(self): st = p.stat() self.assertEqual(st, p.stat(follow_symlinks=False)) - @needs_symlinks - def test_lstat(self): - p = self.cls(self.base)/ 'linkA' - st = p.stat() - self.assertNotEqual(st, p.lstat()) - - def test_lstat_nosymlink(self): - p = self.cls(self.base) / 'fileA' - st = p.stat() - self.assertEqual(st, p.lstat()) - def test_is_dir(self): P = self.cls(self.base) self.assertTrue((P / 'dirA').is_dir()) @@ -2868,11 +2856,13 @@ def test_delete_dir(self): base = self.cls(self.base) base.joinpath('dirA')._delete() self.assertRaises(FileNotFoundError, base.joinpath('dirA').stat) - self.assertRaises(FileNotFoundError, base.joinpath('dirA', 'linkC').lstat) + self.assertRaises(FileNotFoundError, base.joinpath('dirA', 'linkC').stat, + follow_symlinks=False) base.joinpath('dirB')._delete() self.assertRaises(FileNotFoundError, base.joinpath('dirB').stat) self.assertRaises(FileNotFoundError, base.joinpath('dirB', 'fileB').stat) - self.assertRaises(FileNotFoundError, base.joinpath('dirB', 'linkD').lstat) + self.assertRaises(FileNotFoundError, base.joinpath('dirB', 'linkD').stat, + follow_symlinks=False) base.joinpath('dirC')._delete() self.assertRaises(FileNotFoundError, base.joinpath('dirC').stat) self.assertRaises(FileNotFoundError, base.joinpath('dirC', 'dirD').stat)

1 0

[3.13] gh-127316: fix incorrect assertion in setting `__class__` in free-threading (GH-127399) (#127422)
by kumaraditya303 Nov. 29, 2024

Nov. 29, 2024

https://github.com/python/cpython/commit/2f56c68dec97002fdd8563a0e4977b75eb… commit: 2f56c68dec97002fdd8563a0e4977b75eb191ab9 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: kumaraditya303 <kumaraditya(a)python.org> date: 2024-11-29T16:36:44Z summary: [3.13] gh-127316: fix incorrect assertion in setting `__class__` in free-threading (GH-127399) (#127422) gh-127316: fix incorrect assertion in setting `__class__` in free-threading (GH-127399) (cherry picked from commit 45c5cba318a19dda3ee6f9fc84781cc7a2fbde80) Co-authored-by: Kumar Aditya <kumaraditya(a)python.org> files: M Lib/test/test_free_threading/test_type.py M Objects/dictobject.c diff --git a/Lib/test/test_free_threading/test_type.py b/Lib/test/test_free_threading/test_type.py index 3e565cb7ea0f7b..d847acfd6f9edb 100644 --- a/Lib/test/test_free_threading/test_type.py +++ b/Lib/test/test_free_threading/test_type.py @@ -125,6 +125,21 @@ def work(): for thread in threads: thread.join() + def test_object_class_change(self): + class Base: + def __init__(self): + self.attr = 123 + class ClassA(Base): + pass + class ClassB(Base): + pass + + obj = ClassA() + # keep reference to __dict__ + d = obj.__dict__ + obj.__class__ = ClassB + + def run_one(self, writer_func, reader_func): writer = Thread(target=writer_func) readers = [] diff --git a/Objects/dictobject.c b/Objects/dictobject.c index fccc8e930f51d6..f1f9110ff73e6d 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -7190,7 +7190,7 @@ _PyDict_DetachFromObject(PyDictObject *mp, PyObject *obj) // We could be called with an unlocked dict when the caller knows the // values are already detached, so we assert after inline values check. - _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(mp); + ASSERT_WORLD_STOPPED_OR_OBJ_LOCKED(mp); assert(mp->ma_values->embedded == 1); assert(mp->ma_values->valid == 1); assert(Py_TYPE(obj)->tp_flags & Py_TPFLAGS_INLINE_VALUES);

1 0

[3.13] Link to correct class methods in asyncio primitives docs (GH-127270) (#127423)
by kumaraditya303 Nov. 29, 2024

Nov. 29, 2024

https://github.com/python/cpython/commit/ee57221872ad009df648e22d2f113f9076… commit: ee57221872ad009df648e22d2f113f907607ec16 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: kumaraditya303 <kumaraditya(a)python.org> date: 2024-11-29T16:26:42Z summary: [3.13] Link to correct class methods in asyncio primitives docs (GH-127270) (#127423) Link to correct class methods in asyncio primitives docs (GH-127270) (cherry picked from commit 15d6506d175780bb29e5fcde654e3860408aa93e) Co-authored-by: Ollanta Cuba Gyllensten <ollantaster(a)gmail.com> files: M Doc/library/asyncio-sync.rst diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst index 3cf8e2737e85dc..77c2e97da11990 100644 --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -259,8 +259,8 @@ Condition Note that a task *may* return from this call spuriously, which is why the caller should always re-check the state - and be prepared to :meth:`wait` again. For this reason, you may - prefer to use :meth:`wait_for` instead. + and be prepared to :meth:`~Condition.wait` again. For this reason, you may + prefer to use :meth:`~Condition.wait_for` instead. .. coroutinemethod:: wait_for(predicate) @@ -268,7 +268,7 @@ Condition The predicate must be a callable which result will be interpreted as a boolean value. The method will repeatedly - :meth:`wait` until the predicate evaluates to *true*. The final value is the + :meth:`~Condition.wait` until the predicate evaluates to *true*. The final value is the return value. @@ -434,7 +434,7 @@ Barrier .. coroutinemethod:: abort() Put the barrier into a broken state. This causes any active or future - calls to :meth:`wait` to fail with the :class:`BrokenBarrierError`. + calls to :meth:`~Barrier.wait` to fail with the :class:`BrokenBarrierError`. Use this for example if one of the tasks needs to abort, to avoid infinite waiting tasks.

1 0